1 Measure/probability spaces

Some guidance

This first chapter is very long, sorry! This is in particular due to the fact that there is a lot of revision of stuff you will already know in Section 1.2 and Section 1.3 (so obviously skimming through to convince yourself you still have that all sharp is of course sufficient!). If you find this daunting, then I have two tips:

Recall that in List of all definitions, results and examples you can find an overview of all definitions, results and examples. You could have a look there first to pick out the things that are new to you for instance, and to get a good compact overview of the different topics that are discussed in this chapter.
There are four conceptually really key examples in this chapter: Example 1.48, Example 1.52, Example 1.54, and Example 1.58. If nothing else, then please do make sure to read and study these carefully!

1.1 Why??

Once upon a time (I’m imagining) people were spending their days hunting mammoths and collecting berries, and in the evening they would regularly sit down for a beer and a nice little game of rolling some dice. Of course, at least earlier in the evening when there wasn’t too much beer in the men yet, the players would try to gain a little advantage by trying to work out how likely certain outcomes were. Straightforward and very intuitive to compute because there are only finitely many, all equally likely, possible outcomes in a dice game so you simply count the number of outcomes you are interested in and divide by the total number of outcomes. That was the whole of Probability Theory for you — nice and simple!

But then mathematicians came along who desperately wanted to generalise these concepts — to spoil the fun for everybody and make it all a lot harder. :(. Or maybe, to be fair, they actually had some very good reasons for doing so. For example, numbers (integers) came about organically as a tool to count things. Five berries, three mammoths, that type of stuff. Mathematicians took that idea and generalised it to the set of natural numbers $\N=\{0,1,2,\ldots\}$. Although this is a relatively straightforward generalisation it still is one, for instance because $\N$ has infinitely many elements i.e. there is no largest integer. From a practical counting perspective that doesn’t seem to make a lot of sense, I mean, do you really need truly massive integers? Surely you can take some large integer, like the number of atoms in the universe or something, and don’t bother with all integers larger than that one — after all you can’t possibly collect more of anything than there are atoms in the universe so we can just stop with our integers there? While that is not false of course, the fact that we have defined our set of integers as the infinite set $\N$ and not just only the ones we may need for counting purposes is absolutely crucial to pretty much all of modern mathematics!

A similar thing is true for Probability Theory. Yes we could very well have left things at random experiments with only finitely many outcomes — that would have been plenty to deal with any dice rolling game, and arguably to deal with any random experiment you can actually practically execute in the real world. But then think back to your basic probability courses and one of the most important and impressive results you encountered there: the Central Limit Theorem. Think back for a moment how often you have used it (maybe unknowingly!) to (approximately) compute something that would have been at least a massive pain or even completely impossible to compute without the CLT. And then think about that the formulation of the CLT needs the Normal distribution (which of course has many more important uses than just the CLT) which doesn’t at all exist in the world of random experiments with only finitely many outcomes¹!

¹ Can you explain why?

Bottom line: if mathematicians wouldn’t have generalised Probability Theory beyond these basic, real world experiments with only finitely many possible outcomes to a more general framework, then we would never have known Normal distributions nor the CLT! And of course the Normal distribution and the CLT are only quite basic examples of the many achievements of modern Probability Theory building on that more general framework — in this course we will talk about stochastic processes and in particular martingales as fascinating examples of what is all out there.

Modern Probability Theory takes the intuitive understanding we all (well almost all maybe judging by the people taking part in lotteries and things ;)) have about likelihoods/probabilities of outcomes/events in basic experiments and generalises that intuition to a very rich mathematical framework. Judging by the number of mathematicians all over the world working in this area as well as its many applications to an immense variety of real world problems (finance, insurance, engineering, biology, physics, chemistry, sports science, …) the development of modern Probability Theory is arguably one of the most important and impressive achievements in mathematics in modern times.

But yeah… it does need some work on our part to get into it! And so, before we can start talking about stochastic processes and martingales and other fascinating objects living in that world (from Week 5 onwards), we will use the first four weeks of this course to give you a crash course into the necessary basis: the building blocks of modern (i.e. measure theoretic) Probability Theory.

During this first week we don’t encounter much probability related stuff just yet, we first need to introduce some measure theoretic concepts before we can start exploring this exciting world of modern Probability Theory! We do a bit of long run up, starting by revisiting some more general, basic mathematical concepts that we will use a lot throughout (Section 1.2 and Section 1.3). Then in Section 1.4 we come back to the surface to talk a bit more precisely about the journey we’re about to embark upon, and in Section 1.5 we get started for real!

1.2 A tiny drip of mathematical logic

Just a tiny drip, basically just to write down two famous logical proof techniques that should be in the toolbox of any mathematician, and in fact quite near the top because they’re quite often useful! As we all know, ultimately what makes mathematics so incredibly powerful is that our arguments (i.e. computations/proofs) form a logical chain that brings us from our assumptions to our conclusion in an absolutely inescapable way: if the assumptions hold, then the conclusion must hold as well. No ifs, buts, depends on the weather, what nonsense Donald Trump burbs about it, or what your favourite Tiktok influencer thinks about it. Cold hard truth!

The smallest (“atomic”) parts that make up such a logical chain from assumptions to ultimate conclusion come in essentially only two flavours: implications (symbol: $\implies$) and equivalences (symbol: $\iff$). In fact, any key result (no matter whether that’s formulated as a theorem or a lemma or a proposition or …) is normally either an implication or an equivalence. Let’s recall quickly: if $P$ and $Q$ are two (mathematical) statements, each of which is either true or false, then these two flavours are the following two relationships between $P$ and $Q$:

$P \implies Q$ (“$P$ implies $Q$”): this is the following relationship: if $P$ is true, then $Q$ is true as well;
$P \iff Q$ (“$P$ is equivalent to $Q$”): this is the following relationship: $P$ is true if and only if $Q$ is true.

Of course, for any choice of $P$ and $Q$, either of these two relationships may or may not hold! Indeed, a typical interesting theorem looks like “If [condition(s)] hold, then we have that [conclusion(s)]” i.e. the theorem is the statement that the relationship $P \implies Q$ actually holds (where $P$ are the assumption(s), and $Q$ the conclusion(s)). And then of course, what makes an interesting theorem interesting is that it is not at all obvious that that implication holds, it establishes a useful/interesting previously unknown implication!

The proof of such a theorem then normally consists of a number of other statements in addition to $P$ and $Q$, say $L_1$, $L_2$, …, $L_n$ that gradually lead from $P$ to $Q$, plus arguments that show that each of the following relationships hold: \[P \implies L_1, \quad L_1 \implies L_2, \quad \ldots, \quad L_n \implies Q.\] If all these relationships are shown to hold, then it also follows that the relationship $P \implies Q$ holds (transivity). This is of course in more detail the “logical chain that brings us from our assumptions to our conclusion” we mentioned above!

The same story applies to chains of equivalences, but recall that equivalence can also be broken down into two implications: the relationship $P \iff Q$ holds exactly if the relationships $P \implies Q$ and $Q \implies P$ both hold (and hence, the relationship $P \iff Q$ fails to hold exactly if at least one of two relationships $P \implies Q$ and $Q \implies P$ fails to hold). So if a theorem states that an equivalence between two statements holds, then the proof may well consist of two (chains of) implications.

Example 1.1 (Some examples of implications and equivalences) Here are some simple examples to see the above two relationships in action, where $x$ is (positive or negative) integer:

$P$	$Q$	$P \implies Q$	$P \iff Q$
$x=-2$	$x^2=4$	holds/true	doesn’t hold/false
$x^2=4$	$x=-2$	doesn’t hold/false	doesn’t hold/false
$x=-2$ or $x=2$	$x^2=4$	holds/true	holds/true

Observe that on the second row, if $x^2=4$ (i.e. $P$ is true), then it follows that $x=-2$ or $x=2$. It is not guaranteed that $x=-2$, so it is not guaranteed that $Q$ is true, and so the relationship $P \implies Q$ fails to hold. For the $P \iff Q$ column, if you’re not sure then just apply what we said just above this example: $P \iff Q$ holds exactly if $P \implies Q$ and $Q \implies P$ both hold.

Of course, you are already quite a competent mathematician, so none of the above will come as a surprise! As mentioned above, I just wanted to quickly reiterate it to highlight two particularly useful proof/logic techniques. After all, if we want to prove some neat theorem then as mentioned above, we need to establish a chain of implications/equivalences that take us from the assumptions to the conclusion of the theorem, and in particular we need to show that each of the implications/equivalences in our chain is actually valid i.e. is a relationship that actually holds!

Sometimes this is simple enough by “direct” reasoning. For instance if $P$ is the statement that $x=-2$ and $Q$ the statement that $x^2=4$, then a proof that the relationship $P \implies Q$ holds is simply to say: if $x=-2$ then $x^2=(-2)^2$ i.e. $x^2=4$. But obviously more interesting implications are often not as easy to prove, and in such cases it is very good to be aware that we have two neat logic tricks up our sleeve that we can always use:

Definition 1.2 (Proof by contraposition and by contradiction) Let $P$ and $Q$ be any (mathematical) statements. Then:

To prove that $P \implies Q$ holds by “direct” reasoning, you simply start by assuming that $P$ is true and come up with (legal, ideally ;)) steps to conclude that then $Q$ must also be true.
To prove that $P \implies Q$ holds, the principle of proof by contraposition is that we may equivalently also prove that the following relationship holds: $\text{not } Q \implies \text{not } P$.
To prove that $P \implies Q$ holds, the principle of proof by contradiction is that we assume that the relationship $P \implies Q$ does not hold (i.e. that it is possible for $P$ to be true and $Q$ to be false) and come up with steps to show that that assumption leads to a contradiction i.e. a conclusion that we know to be false.
To prove that $P \iff Q$ holds, it is often easiest to use that this is equivalent to $P \implies Q$ and $Q \implies P$, and prove that both these relationships hold (more or less) seperately from each other, using one of i–iii above for instance.

Example 1.3 (Examples of proof by contraposition and by contradiction)

First let’s do one by contraposition. Suppose that we want to prove the following implication (where $n$ is a positive integer): if $n^2$ is odd ($P$), then $n$ is odd ($Q$). Applying proof by contraposition we change tracks to proving the implication: if not $Q$ then not $P$ i.e. if $n$ is even then $n^2$ is even (where we used that any positive integer is either even or odd). And this final implication is quite easy to show: assume that $n$ is even. Then we can write it as $n=2a$ for some integer $a$, and hence $n^2=(2a)^2=4a^2=2 \cdot 2a^2$. So $n^2$ is divisible by $2$ and therefore indeed even.
Next one by contradiction. Again within the positive integers, consider the implication: if $5n+2$ is odd ($P$), then $n$ is odd ($Q$). To prove this via contradiction, assume that it is possible that $P$ and not $Q$ both hold i.e. that there exists a positive integer $n$ so that $5n+2$ is odd ($P$) and $n$ is even (not $Q$). Now we are looking to find a contradiction (any contradiction will do) in this situation. For example: since $n$ is even we can write it as $n=2a$ for some integer $a$. Then $5n+2=5(2a)+2=10a+2=2 \cdot (5a+1)$ i.e. $5n+2$ is divisible by $2$ and therefore even. But this contradicts with $P$.

Now, these are of course just some quick and dirty toy examples — you may well know much more interesting examples yourself. The key point is: if you need to prove an implication and are running out of ideas/don’t see an obvious strategy for how to do this “directly”, then it’s worth to take a look at whether a proof by contraposition or by contradiction is more appealing!

Remark 1.4. Of course, proof by contradiction can be applied to any statement, not only to implications as we formulated in Definition 1.2. For instance, consider the statement “there exists no largest integer”. To prove this by contradiction, we assume that this statement is false, i.e. that it holds that there does exist a largest integer, and derive a contradiction. For instance as follows: call this largest integer $N$. Now consider $N+1$. Clearly this is also an integer, and larger than $N$. But this contradicts with the assumption we just made that $N$ is the largest integer. Done.

1.3 Sets, and sets of sets, and …

Next, after the bit of logic above, this section is again devoted to a concept you’re already quite familair with, namely good old sets. They will pop up an awful lot in this course (e.g. as events in our Probability context) and therefore I think it is worth to quickly run through the basics, also to make sure that the notation we’ll be using is crystal clear for everybody!

1.3.1 Just sets

Let’s start from scratch. Recall that mathematically, a set is just a bag/container whose only purpose in life is to hold a bunch of things that we call its elements. Can be numbers, but also functions or matrices or dog biscuits or whatever you want (even other sets!). That’s all — a set does not (need to) have any further structure.

The basic mathematical notation for for instance a set $\Omega$ containing the numbers $1$, $2$ and $4$ is to write: $\Omega=\{1,2,4\}$. Or, equivalently, $\Omega=\{4,1,2\}$ for instance — a set does not care about order of elements or stuff like that, it’s really only about which elements it contains. If we define the function say $f(x)=4x^2$, then for instance $\Omega=\{f,2,\{5,10\}\}$ is also a perfectly valid example of a set, containing three elements: the function $f$, the number $2$, and the set $\{5,10\}$.

Further, to express “is an element of (a set)” resp. “is not an element of (a set)” we use the symbol $\in$ resp. $\not\in$. For example, with $\Omega=\{1,2,4\}$ it holds that $1 \in \Omega$ and also that $10 \not\in \Omega$. If we take the example $\Omega=\{f,2,\{5,10\}\}$ from the previous paragraph, then it holds that $\{5,10\} \in \Omega$ (after all, $\{5,10\}$ is an element of $\Omega$, irrespective of whether it is a set on its own!) and $5 \not\in \Omega$ (because the number $5$ is not one of the three elements of $\Omega$ — the fact that it happens to be an element of $\{5,10\}$ is irrelevant!).

A slightly extended form of that basic notation to define a set that we commonly use is so-called “set-builder notation”, which expresses the sentence “$\Omega$ is the set consisting of all elements $x$ from [some (other) set] that satisfy [condition]” in mathematical notation as follows²: \[\Omega=\{ x \in \text{[some (other) set]} \, | \, \text{[condition on } x \text{]} \}.\] For instance, to define the set $\Omega$ consisting of all integers from $1$ to $10$ that are even we could for instance write \[\Omega=\{ x \in \{1,2,\ldots,10\} \, | \, x \text{ is even} \}. \tag{1.1}\] Of course in this simple case it is easy to figure out all the elements of $\Omega$ and we could equally well use the basic notation $\Omega=\{2,4,6,8,10\}$, but in more complicated cases that “set-builder notation” can be very handy!

² In this course we’ll use a “|” in this notation because that’s how I was raised ;), but you also regularly see people using a “:” rather

Remark 1.5. Note that in the notation Equation 1.1, the variable $x$ is a “dummy variable” that has no meaning outside the expression Equation 1.1. We just need some variable in that expression to make the notation work, so we have to use something, but it could equally well have been anything other than $x$. Other well known cases in maths where such dummy variables are used:

When you define a function, what you actually define is a rule that describes how the function acts on its argument(s) i.e. how the function turns input into output. It’s just that we need some variable to express the rule in mathematical notation, but that is then a dummy variable. For instance the function that takes one argument and then squares it, could equally well be mathematically expressed in any of the forms \[f(x)=x^2, \quad f(y)=y^2, \quad f(\text{Kevin})=\text{Kevin}^2, \quad \ldots.\]
When you integrate a function $f$ over some interval $[a,b]$ you also use a dummy variable to express this operation: you could equally well write it as \[\int_a^b f(x) \d x, \quad \int_a^b f(u) \d u, \quad \int_a^b f(\text{Kevin}) \d \text{Kevin}, \quad \ldots.\]

For those of you with some coding experience, dummy variables exist in very similar fashion there as well: in a loop like

for i in ... {
  do something with i
}

the variable i is a dummy variable that only exists inside this loop and could equally well have been replaced by any other variable/symbol (as long as it hasn’t been defined/used yet typically).

Next let’s quickly recall the sets that are so prominent that they have their very own dedicated symbol:

Definition 1.6 (Standard sets: empty, integers, rational numbers and real numbers)

$\emptyset$ is the notation we use for the empty set. It represents the set that contains no elements at all.
The set of natural numbers/non-negative integers is denoted by $\N$: $\N=\{0,1,2,\ldots\}$.
The set of all integers is denoted by $\Z$: $\Z=\{\ldots,-2,-1,0,1,2,\ldots\}$.
The set of all rational numbers is denoted by $\Q$ i.e. $\Q$ contains those numbers that can be expressed as a fraction $p/q$ for integers $p,q$ with $q \not= 0$. For example, the number $-10.5$ can also be written as $-105/10$ and is therefore an element of $\Q$. On the other hand, for instance the famous contants $\pi=3.141...$ and $e=2.718...$ cannot be expressed in this fraction form and are therefore not elements of $\Q$ (they are therefore called, indeed, irrational numbers).
The set of all real numbers is denoted by $\R$. A bit loosely speaking, anything that we typically understand to be a “number” i.e. anything that can be expressed as a finite decimal expansion (like $2$, or $0.5$, or $-4.561$) or infinite decimal expansion (like $\pi$ or $e$).

As long as we work in the context of integers or rational numbers, defining sets by listing its elements (e.g. $\Omega=\{1,2,4\}$) makes perfectly good sense. However within the real numbers this is too limited — many interesting sets in the context of the real numbers are “too large/contain too many elements” (cf. Subsection 1.3.2) to be able to express them in this way. In those cases, we use interval notation rather:

Definition 1.7 (Sets on the real line: intervals) Choosing any $a \in \R$ (or $a=-\infty$) and $b \in \R$ (or $b=\infty$) with $a \leq b$, there are four variations to capture the set of all real numbers between $a$ and $b$ (intervals), depending on whether $a$ and $b$ themselves are included or not: \[\begin{align*} (a,b) &= \{ x \in \R \, | \, a<x<b \}, \\ [a,b) &= \{ x \in \R \, | \, a \leq x<b \}, \\ (a,b] &= \{ x \in \R \, | \, a<x \leq b \}, \\ [a,b] &= \{ x \in \R \, | \, a \leq x \leq b \}. \end{align*} \]

As we said earlier, a set is just a container holding elements, that’s it. There is then, as you know well, also not too much in the way of relationships between sets³:

³ Again an issue with being properly raised here ;): some people like to use “$\subset$” to mean “is a subset of” i.e. where we use a “$\subseteq$”

Definition 1.8 (Relationships between sets) Let $A, B$ be any sets. Then we say that

$A$ is equal to $B$ i.e. $A=B$ holds if and only if they contain exactly the same elements (i.e.: $x \in A \iff x \in B$);
$A$ is a subset of $B$ i.e. $A \subseteq B$ holds if and only if each element in $A$ is also an element of $B$ (i.e.: $x \in A \implies x \in B$);
$A$ is a strict subset of $B$ i.e. $A \subset B$ holds if and only if each element in $A$ is also an element of $B$ and the sets are not equal (i.e.: $A \subseteq B$ and $A \not= B$).

Note that by convention, the empty set is a subset of any other set i.e. $\emptyset \subseteq A$ is always true, no matter what set $A$ is. Kind of similar to how $0$ is less than or equal to any non-negative integer if you will. Those of you who are fond of their mathematical logic might appreciate that this convention is consistent with the above mathematical definition of “is subset of”!

Remark 1.9. Observe that for any two sets $A$ and $B$, $A=B$ holds if and only if both $A \subseteq B$ and $B \subseteq A$ hold. This can be handy to use in a situation where $A$ and $B$ are complicated sets and you want to prove that $A=B$, because this observation shows that one option is to this in two steps:

Show $A \subseteq B$. Here you start with “Let $x \in A$”, followed by (ideally valid :)) arguments to be able to conclude “Hence it follows that $x \in B$”.
Show $B \subseteq A$. Here you go the other way around: starting with “Let $x \in B$”, you look for arguments to be able to conclude “Hence it follows that $x \in A$”.

Indeed this is structurally very similar to how an equivalence can equally well be seen as two implications, as we mentioned in Definition 1.2!

Next let’s quickly remind ourselves of the standard set operations. For this, it is handy to consider sets in the context of some “master set” in which all elements that we may or may not want to use for our sets are contained. For instance, if we want to work with integers it makes sense to choose $\N$ (or $\Z$ if we want negative integers as well), or if we’re interested in real numbers than it’s naturally $\R$. We call such a “master set” (appropriately) our universe.

Definition 1.10 (Operations on sets) Let $\Omega$ be a universe and let $A, B \subseteq \Omega$. Then we have the following operations to construct new sets (in $\Omega$).

$A^c$: the complement of $A$ is the set containing all elements in $\Omega$ that are not elements of $A$ i.e. \[A^c=\{ x \in \Omega \, | \, x \not\in A \}.\]
$A \cup B$: the union of $A$ and $B$ is the set containing all elements in $\Omega$ that are either an element of $A$ or an element of $B$ (or both) i.e. \[A \cup B=\{ x \in \Omega \, | \, x \in A \text{ or } x \in B \}.\]
$A \cap B$: the intersection of $A$ and $B$ is the set containing all elements in $\Omega$ that are an element of $A$ as well as an element of $B$ i.e. \[A \cap B=\{ x \in \Omega \, | \, x \in A \text{ and } x \in B \}.\]
$A \setminus B$: the set difference of $A$ and $B$ is the set containing all elements in $\Omega$ that are an element of $A$ but not an element of $B$ i.e. \[A \setminus B=\{ x \in \Omega \, | \, x \in A \text{ and } x \not\in B \}.\]

Some notes:

Unions/intersections of sets are similar operations as adding up/multiplying numbers for instance: if you can do it with two, you can do it with any number of operands by just repeating the process multiple times. For instance, $A \cup B \cup C$ is naturally given by $(A \cup B) \cup C$ (so you first take the union of the pair $A$ and $B$, and then the union of that result with $C$). Extending this thinking, if $A_1, A_2, \ldots, A_n$ are all subsets of $\Omega$, then we can naturally talk about their union and their intersection: \[\bigcup_{i=1}^n A_i = \{ x \in \Omega \, | \, \text{there exists an } i \in \{1,\ldots,n\} \text{ so that } x \in A_i \} \tag{1.2}\] and \[\bigcap_{i=1}^n A_i = \{ x \in \Omega \, | \, x \in A_i \text{ for all } i \in \{1,\ldots,n\} \} \tag{1.3}\] (observe that in Equation 1.2 we throw all elements from the $A_i$’s together on a pile to create one new large set out of them, while in Equation 1.3 we fleece through all the $A_i$’s to find the elements that are present in each of them and keep only those).

And even further, if we have an infinite sequence of subsets $A_1, A_2, \ldots$ of $\Omega$: \[\bigcup_{i=1}^\infty A_i = \{ x \in \Omega \, | \, \text{there exists an } i \in \{1,2,\ldots\} \text{ so that } x \in A_i \} \tag{1.4}\] and \[\bigcap_{i=1}^\infty A_i = \{ x \in \Omega \, | \, x \in A_i \text{ for all } i \in \{1,2,\ldots\} \}. \tag{1.5}\]
There are a whole bunch of well known identities involving these set operations, see e.g. Wiki for a reminder if you’d like. Recall that Venn diagrams are often a useful way to visualise these operations and identities. One such identity that is worth recalling explicitly are the De Morgan’s laws: \[(A \cup B)^c = A^c \cap B^c \quad \text{and} \quad (A \cap B)^c = A^c \cup B^c,\] which can be generalised from two to arbitrarily many sets (see e.g. Section 1.2 in Grimmett and Stirzaker 1992).

Example 1.11 (Infinite unions and intersections) Working in the universe $\Omega=\R$ with interval notation from Definition 1.7, let’s do some quick examples.

Set $A_i=(-i,i)$ for $i=1,2,\ldots$, then \[\bigcup_{i=1}^\infty A_i=\R.\] Why? Intuitively because these intervals $(-i,i)$ get ever larger as $i$ grows, ultimately filling up the whole real line. For a mathematical argument, let’s follow the strategy laid out in Remark 1.9.
- $\cup_{i=1}^\infty A_i \subseteq \R$: that’s clear because we always stay inside our universe $\Omega=\R$.
- $\R \subseteq \cup_{i=1}^\infty A_i$: take any $x \in \R$. Clearly for any $i \geq \lvert x \rvert$ we have that $x \in (-i,i)$ and hence $x \in A_i$. Therefore by def (cf. Equation 1.4) $x \in \cup_{i=1}^\infty A_i$.
Set $A_i=(-1/i,1/i)$ for $i=1,2,\ldots$, then \[\bigcap_{i=1}^\infty A_i=\{0\}.\] Why? Intuitively, these intervals $(-1/i,1/i)$ shrink ever further towards the singleton $\{0\}$ as $i$ grows, and hence there is only one element of $\R$ that is an element of all of the $A_i$’s, namely $0$. Verifying this mathematically, again following Remark 1.9:
- $\cap_{i=1}^\infty A_i \subseteq \{0\}$: we need to show that if $x \in \cap_{i=1}^\infty A_i$, then $x \in \{0\}$. Using proof by contraposition (cf. Definition 1.2), we may equally well show that if $x \not\in \{0\}$, then $x \not\in \cap_{i=1}^\infty A_i$. So take $x \not\in \{0\}$ i.e. $x$ is a real number unequal to $0$. Then for all $i$ large enough (in particular, $i>1/\lvert x \rvert$) we have that $x \not\in (-1/i,1/i)=A_i$. So it is not true that $x \in A_i$ for all $i=1,2,\ldots$, and therefore by def (cf. Equation 1.5) $x$ is not an element of $\cap_{i=1}^\infty A_i$, as we wanted.
- $\{0\} \subseteq \cap_{i=1}^\infty A_i$: let $x \in \{0\}$ i.e. $x=0$. Then clearly for any $i=1,2,\ldots$, $x \in (-1/i,1/i)=A_i$. Hence by def (cf. Equation 1.5) also $x \in \cap_{i=1}^\infty A_i$.
Set $A_i=(0,1/i)$ for $i=1,2,\ldots$, then \[\bigcap_{i=1}^\infty A_i=\emptyset.\] Similar to ii above, an ever shrinking collection of intervals — but now there is no real number at all that is an element of all them! A proof analogue to above:
- $\bigcap_{i=1}^\infty A_i \subseteq \emptyset$: we can do this one for instance using proof by contradiction (cf. Remark 1.4). So let’s assume that $\bigcap_{i=1}^\infty A_i \not\subseteq \emptyset$ and work towards a contradiction. This assumption means that there does exist a real number $x$ that is an element of $\bigcap_{i=1}^\infty A_i$. By def (cf. Equation 1.5) this means that $x \in A_i=(0,1/i)$ for all $i=1,2,\ldots$. Clearly $x \leq 0$ immediately leads to a contradiction. So consider $x>0$. Then for any $i>1/x$ we have $1/i<x$ and hence $x \not\in (0,1/i)$, also a contradiction.
- $\emptyset \subseteq \bigcap_{i=1}^\infty A_i$: this is trivially true, recall from Definition 1.8.
Set $A_i=[1,1+1/i)$ for $i=1,2,\ldots$, then \[\bigcap_{i=1}^\infty A_i=\{1\}.\]
Set $A_i=[-2+1/i,2-1/i]$ for $i=1,2,\ldots$, then \[\bigcup_{i=1}^\infty A_i=(-2,2).\]

Leaving the intuition and proof of iv and v for you to think about, cf. Exercise 1.1! :).

We have one more, kind of, set operation to discuss in this section. You’ll recall quite well that $\R^2$ denotes the set of all pairs of real numbers: $(x,y)$ for $x,y \in \R$, which we often visualise as “the plane” with a horizontal and a vertical axis and things. More generally, for any integer $n=1,2,\dots$, $\R^n$ is the set of all sequences of length $n$ (or “$n$-tuples”) of real numbers: $(x_1,x_2,\ldots,x_n)$ where $x_i \in \R$. These are some well known examples of Cartesian products:

Definition 1.12 (Cartesian products) If $A$ and $B$ are any two sets, then the Cartesian product of $A$ and $B$ is the set denoted by $A \times B$ whose elements are all possible pairs with the first entry from $A$ and the second from $B$: \[A \times B = \{ (a,b) \, | \, a \in A \text{ and } b \in B \}.\] We can naturally extend this definition to a Cartesian product of three or more sets.

As a special case, we can take the Cartesian product of a set $A$ with itself for which we then conveniently use “power notation”: $A^2 := A \times A$, $A^3 := A \times A \times A$ etc. In general, for any integer $n=2,3,\ldots$, $A^n$ hence denotes the “$n$-fold Cartesian product with itself” i.e. the set whose elements are all possible sequences of length $n$ filled with entries from $A$: \[A^n := \{ (a_1,a_2, \ldots,a_n) \, | \, a_i \in A \text{ for all } i \in \{1,\ldots,n\} \}.\] Sometimes you also this notation naturally extended to such sequences of infinite length: \[A^\infty := \{ (a_1,a_2, \ldots) \, | \, a_i \in A \text{ for all } i \in \{1,2,\ldots\} \}.\]

Finally in this section, let’s briefly revisit the concept of open and closed sets. Fundamentally, universes that allow for a natural definition of open and closed subsets (and the many consequences that flow from it) come in a wild variety and are collectively studied in a field called topology. For us, it is sufficient to briefly revisit the situation on the real line:

Definition 1.13 (Open and closed subsets of the real line) Working in the universe $\Omega=\R$:

A subset $A \subseteq \R$ is called
- open if for any $x \in A$ there exists a $h>0$ so that $(x-h,x+h) \subseteq A$,
- closed if for any convergent (with finite limit) sequence $(x_i)_{i=1,2,\ldots}$ with entries from $A$, its limit is also an element of $A$.
$A \subseteq \R$ is open (resp. closed) if and only if $A^c$ is closed (resp. open).

Some notes:

Of course, $\R$ is also a subset of $\Omega=\R$, and as is clear from the above definitions in i it has the special property that it is both open and closed. Since $\emptyset =\R^c$, it follows from ii above that $\emptyset$ is also both open and closed (in a way, this is a convention, a natural one so that the statement ii above is valid for $A=\emptyset$ and $A=\R$ as well).

These are the only two subsets with that special property though, any other subset is exactly one of three options: open, closed, or neither(cf. Example 1.14 below).
Openness has the convenient property that it survives unions: if you take any (finite or infinite) collection of open sets, then their union is also guaranteed to be open (this is straightforward to prove). This is also true for finite intersections, but (maybe surprisingly!) not necessarily for infinite intersections — indeed Example 1.11 ii shows an example of this: we take there an intersection of open intervals but the result is a singleton which is closed (cf. Example 1.14 below).

Example 1.14 (Open and closed subsets of the real line) Looking at some examples of subsets in the universe $\Omega=\R$, using the interval notation from Definition 1.7:

$(a,b)$ is open (also if $a=b$ because then $(a,b)=\emptyset$),
$[a,b]$ is closed (and hence so is any singleton $\{x\}$, since $\{x\}=[x,x]$),
$(a,b]$ and $[a,b)$ for $a<b$ are neither open nor closed (for $a=b$ they both equal $\emptyset$).

If you’d like to see an argument for the final case, take $(a,b]$ for some $a<b$ and apply the criteria from Definition 1.13 i. Firstly, $b \in (a,b]$, but for any for any $h>0$ we see that $(b-h,b+h) \not\subseteq (a,b]$ (after all, $b+h>b$) so $(a,b]$ is not open. Secondly, consider the sequence $(x_i)_{i=1,2,\ldots}$ given by $x_i=a+(b-a)/i$. Then $x_i \in (a,b]$ for all $i=1,2,\ldots$ but the limit of this sequence is $a$ and $a \not\in (a,b]$, so $(a,b]$ is not closed either.

Exercises

You can now do Exercise 1.1.

1.3.2 Countable vs uncountable

A basic property of any set is the number of elements it contains i.e. its cardinality. For sets with finitely many elements this is hardly a very exciting question, just count them eh. However for sets with infinitely many elements the situation is less straightforward. For a long time, no particular distinction was made here, after all the one infinity is the same as the other infinity no? Well, ehm, actually no indeed, at least when it comes to cardinalities!

In the second half of the 19th century, the slightly mad yet genius mathematician/philosopher Georg Cantor rattled some cages by developing a theory insisting that there are, in fact, quite a few different “infinite sizes” that a set can have! For our purposes, we only need a little bit of that madness/genius, as follows.

Imagine a hotel with rooms that are numbered $0$, $1$, $2$, … (so infinitely many rooms). We call a set countable if it is possible to give every element of the set its own hotel room. Clearly this is possible if we consider a finite set with say $N$ elements, then we only need to use rooms $0$, $1$, …, $N-1$. It is (trivially) also possible for the set $\N$, just put each integer $n \in \N$ in room number $n$. Maybe less obvious, but it is also possible for the set $\Z$: put $0$ in room number $0$, $1$ in room number $1$, $-1$ in room number $2$, $2$ in room number $3$, $-2$ in room number $4$ etc. It is even still possible for the set $\Q$, with a somewhat smart “room allocation rule”.

To translate this setup into maths: a set $A$ is countable if each element of $A$ can be assigned to its own element of $\{0,1,2,\ldots\}$, that is, if there exists an injective (or: one-to-one) function $f: A \to \{0,1,2,\ldots\}$.

Ok. Clearly Cantor wouldn’t have wasted his time on this if any set was countable, then this would all be a moot point. Indeed there do exist sets that are not countable, and these are, indeed, called uncountable. You can think about uncountable as a “larger size” than countable. The most well known example of an uncountable set is the real line $\R$. Of course there is something to prove here! How do you prove that something does not exist, in this case an injective function $f:\R \to \{0,1,2,\ldots\}$? Well indeed, you employ “proof by contradiction” (cf. Example 1.11): you assume that such a function does exist and then work towards a contradiction. That’s the strategy that Cantor used for his famous diagonal argument to establish that $\R$ is uncountable.

Uncountable sets, like $\R$, are dark and dangerous monsters! We as humans all have an ingrained intuition of how things like numbers and sets (should) behave. However this is fundamentally rooted in our experience living in a “finite world” (three cows, five beers, that stuff). This intuition extends quite naturally to working with countable objects, but dealing with uncountable objects is like stepping into a spaceship to some alien planet where many things don’t work as you intuitively expect with your experience from earth. Of course, that is exactly a situation in which mathematics shines due its unbreakable rigour, but it does mean that you have to be very careful when you apply your intuition: double check whether everything you’re doing is indeed mathematically legal!

Remark 1.15. Now, if you haven’t encountered these concept before, you may very well be wondering what all the drama in the paragraph above this remark is about. It is maybe not so very obvious, but here is just one example of the weirdness. Suppose, for instance, that you have a set of numbers $A$ and want to compute the sum of these numbers. Working carefully (keeping in mind the “mathematically legal” comment above), addition is in principle defined for two numbers only (“binary operation”), but can be extended to multiple numbers by applying the definition for the sum of two numbers multiple times (as also mentioned in Definition 1.10): $5+4+3=(5+4)+3=9+3=12$.

So a natural algorithm/rule for computing the sum of all elements in $A$ is as follows. Start with the first two elements and add them up. Then pick the third number and add that one to the subtotal. Then pick the fourth number and add that one to the subtotal as well. Etc.

Maybe you can start smelling the rat now: this is very cute, but it requires that we can actually do stuff like “start with the first two elements”, “then pick the third element” etc. i.e. we need to be able to “list” all our elements one after the other, or in the above hotel analogy: we need to have one number in each of these hotel rooms so that we can visit them one by one by walking through the hotel. That is, it requires that our set $A$ is countable! Indeed, computing the sum of all elements in an uncountable set $A$ is not a well defined operation, simply can’t be done (in a sensible way)!

Definition 1.16 (Countable vs uncountable sets) A set $A$ is called countable if there exists an injective (or: one-to-one) function $f: A \to \{0,1,2,\ldots\}$. This includes any finite set, and also some infinite sets like $\N$, $\Z$ and $\Q$ (such sets are also called countably infinite).

Intuitively, a set $A$ is countable if and only if we are able to list all its elements, one after the other (see the hotel rooms analogy used earlier in this section), i.e. if we are able to express $A$ as $A=\{a_1,\ldots,a_n\}$ (finite) or $A=\{a_1,a_2,\ldots\}$ (countably infinite).

A set $A$ that is not countable is called uncountable.

Example 1.17 (Examples of & rules for countable vs uncountable sets) Here are some more examples of, and “rules” for, countable and uncountable sets. Most of the examples and rules make perfect sense if you keep two things in mind: uncountable is a “larger size” than countable, and if $A \subseteq B$ then naturally $B$ has a “size” at least as large as $A$.

Note: most of this example is fyi, for exam purposes it is sufficient to be aware of points iv and v below.

If $B$ is countable and $A \subseteq B$, then $A$ is also countable.
If $A$ is uncountable and $A \subseteq B$, then $B$ is also uncountable.
If $A$ is finite with $n$ elements resp. countable resp. uncountable and there exists a bijective function (or: one-to-one correspondence) $f: A \to B$, then $B$ is also finite with $n$ elements resp. countable resp. uncountable.
Any subset of $\N$, $\Z$ or $\Q$ is countable (cf. Definition 1.16 and apply rule i).
Any interval $(a,b) \subseteq \R$ with $a<b$ is uncountable. Hence so is $\R$ itself (apply rule ii) and so is any interval $[a,b]$, $(a,b]$ or $[a,b)$ with $a<b$ (since in all three cases, you can construct an open interval $(c,d)$ that is subset of the interval in question and then apply rule ii).
If a set $A$ is uncountable, then so is $A^n$ for any $n=2,3,\ldots$ and also $A^\infty$ (recall the notation from Definition 1.12; this follows from rule ii & iii).
If a set $A$ is countable, then for any integer $n=2,3,\ldots$ the set $A^n$ is also countable. But the set $A^\infty$ is uncountable! (recall again the notation from Definition 1.12).
Fix some sets $C$ and $D$, and let $A$ be the set consisting of all possible functions $f:C \to D$. If $C$ is finite and $D$ is countable, then $A$ is countable. In all other cases, $A$ is uncountable.

Point vii is maybe a bit surprising. Take for example the set $A=\{0,1\}$, which is of course finite with only two elements and therefore countable. For any integer $n=2,3,\ldots$ the set $A^n$ is the set consisting of all possible sequences of length $n$ with entries either $0$ or $1$ (cf. Definition 1.12). Point vii claims this is countable, and this is easy to check: $A^n$ is even finite because it contains $2^n$ elements as I’m sure you’ll agree. Now consider $A^\infty$ i.e. the set of all possible infinite sequences $(a_1,a_2,\ldots)$ with $a_i \in \{0,1\}$. Point vii tells us that somehow, even though for any $n=2,3,\ldots$ the set $A^n$ is finite, the set $A^\infty$ takes a big jump: it is not only infinite but even uncountable! (If you’re interested, this can for instance be proven using Cantor’s diagonal argument).

Finally, an argument for point viii is as follows. First assume that $C$ is countable so that we can list its elements: $C=\{c_1,c_2,\ldots,c_n\}$ or $C=\{c_1,c_2,\ldots\}$. A function $f:C \to D$ is of course nothing but a list of values $f(c_1) \in D$, $f(c_2) \in D$, …, where the list has length $\#C$ (the number of elements in $C$, possibly $\infty$). That is to say: such a function can naturally be associated with a sequence $(d_1,d_2,\ldots,d_n)$ or $(d_1,d_2,\ldots)$ with entries from $D$. In fact, if you think about it, this associaton is a bijection between the set $A$ of all functions $f:C \to D$ on the one hand and the set $B$ of all length $\#C$ sequences with entries from $D$ on the other hand. Now we can apply rule iii to look at the “size” of set $B$ rather, and determine that size by applying rules vi and vii. Finally we can drop the assumption that $C$ is countable by realising that if $C$ is uncountable, then the “size” of $A$ cannot be smaller than when $C$ is countable (intuitively clear I’d guess, but maybe you’d like to think of a rigorous argument yourself!).

Remark 1.18. As we will encounter later, one particular aspect of “countable vs uncountable” that will occasionally be annoying us like a wasp on the beach is in the context of a collection of sets: working in some universe $\Omega$, imagine that we have a subset $A_\alpha \subseteq \Omega$ for each $\alpha \in I$, where $I$ is then some set of numbers called an index set.

Take for instance $\Omega=\R$, and consider the collection, say $\mathcal{C}$, consisting of the subsets/intervals $A_\alpha=[0,e^\alpha] \subseteq \R$ for each $\alpha \in I:=\Z$. So, for each (positive or negative) integer $\alpha$ we have an interval $A_\alpha$. The key point: as we have seen (cf. Definition 1.16) this index set, $\Z$, is countable. This means that $\mathcal{C}$ itself is also countable. Indeed we could write $\mathcal{C}=\{C_1, C_2, \ldots\}$ by for instance choosing $C_1=A_0$, $C_2=A_{-1}$, $C_3=A_{1}$, $C_4=A_2$, $C_5=A_{-2}$, etc. This means that all mathematical operations which require us to “walk through” all the elements of $\mathcal{C}$ are perfectly valid.
Now again take $\Omega=\R$ and let $\mathcal{C}$ consist of all intervals $A_\alpha=[0,e^\alpha] \subseteq \R$ for each $\alpha \in I:=\R$, so we change the index set from the countable $\Z$ to the uncountable $\R$. That’s a perfectly fine definition in principle. However now $\mathcal{C}$ is uncountable (since $I$ is), and hence “walking through” its elements is no longer possible! We saw in Remark 1.15 already an example of a basic operation that suddenly is no longer possible in the face of uncountability. The natural analogue of “adding up all numbers in the set” from that remark is in this context is “taking the union of all subsets in $\mathcal{C}$”. Indeed, this is no longer possible from the perspective of a union as “binary operation”. There is, in this particular case, an escape hatch though: we can take the right hand side of Equation 1.4 and generalise that to still give sensible meaning to “taking the union of all subsets in $\mathcal{C}$”: \[\bigcup_{\alpha \in I} A_\alpha := \{ x \in \R \, | \, \text{there exists an } \alpha \in I \text{ so that } x \in A_\alpha \}\] and similarly for “taking the intersection of all subsets in $\mathcal{C}$” (cf. Equation 1.5): \[\bigcap_{\alpha \in I} A_\alpha := \{ x \in \R \, | \, x \in A_\alpha \text{ for all } \alpha \in I \}.\]

Remark 1.19. Finally it is worth highlighting the relationship between the rational numbers $\Q$ and the real numbers $\R$ a little bit. Clearly $\Q \subseteq \R$, and on the one hand $\R$ is “quite a bit larger” than $\Q$ since $\R$ is uncountable while $\Q$ is countable (cf. Definition 1.16 and Example 1.17). On the other hand, in some sense $\R$ is also not that much larger, since $\Q$ is dense in $\R$.

There’s several (equivalent, of course) ways to formulate what this means, two possibilities are:

For any $x \in \R$, there exists a sequence of rational numbers $(q_n)_{n=0,1,\ldots}$ (i.e. $q_n \in \Q$ for all $n=0,1,\ldots$) so that $\lim_{n \to \infty} q_n=x$.
Fix any $x \in \R$. Then for any $h>0$, no matter how small, the interval $(x-h,x+h)$ contains at least one rational number.

This is actually quite easy to see. For the first one, just write your $x$ in its decimal expansion, and then cut off that expansion after increasingly many digits. For instance, $\pi \in \R$ (recall $\pi=3.141592654...$) can be approximated by the rational numbers $q_0=3$, $q_1=3.1$, $q_2=3.14$, $q_2=3.141$, … Clearly $\lvert \pi-q_n \rvert \leq 10^{-n}$ and hence indeed $\lim_{n \to \infty} q_n=\pi$.

This observation can be very helpful when for some mathematical argument, we actually want to “walk through” the elements of $\R$ but realise that this is impossible due to uncountability. Sometimes we can then still save our poor bums by using that density to show that it is enough to “walk through” $\Q$ rather!

A nice (and maybe somewhat mind blowing for some!) use of this is on show in Exercise 1.2.

Exercises

You can now do Exercise 1.2.

1.3.3 Mutually disjoint sets & partitions

Also this section is about recalling some stuff you will have seen before. You will recall that two sets $A_1$ and $A_2$ are called disjoint if they do not share any elements i.e. if $A_1 \cap A_2=\emptyset$. A natural question is how to generalise this notion to more than two sets. There is more than one way to do that (cf. Remark 1.21), here is the one we want to use:

Definition 1.20 (Mutually disjoint) In some universe $\Omega$, let $A_1, \ldots, A_n$ be a sequence of subsets. We say that they are mutually disjoint if any pair of subsets from that sequence is disjoint. That is, if \[A_i \cap A_j = \emptyset \quad \text{for all } i,j \in \{1,\ldots,n\} \text{ with } i<j. \tag{1.6}\] This extends naturally to an infinite sequence of subsets $A_1, A_2, \ldots$, then the requirement is \[A_i \cap A_j = \emptyset \quad \text{for all } i,j \in \{1,2,\ldots\} \text{ with } i<j. \tag{1.7}\]

Note that Equation 1.6 and Equation 1.7 do indeed cover all possible pairs, they just make use of the observation that if $A_i \cap A_j = A_j \cap A_i$ to reduce the number of checks we actually have to do!

Remark 1.21. Note that $A_1, \ldots, A_n$ being mutually disjoint is a pretty strong requirement, stronger than the requirement \[A_1 \cap A_2 \cap \ldots \cap A_n=\emptyset\] for instance.

Indeed take in $\Omega=\N$ for instance $A_1=\{1,2\}$, $A_2=\{2,3\}$ and $A_3=\{3,4\}$. Then clearly there is no element shared by all three sets i.e. $A_1 \cap A_2 \cap A_3=\emptyset$ (applying Equation 1.3), yet \[A_1 \cap A_2=\{2\} \not= \emptyset \quad \text{and also} \quad A_2 \cap A_3=\{3\} \not= \emptyset,\] so Equation 1.6 definitely does not hold and hence $A_1$, $A_2$, $A_3$ are not mutually disjoint.

Remark 1.22. Just to point out a(n almost) triviality: if you have a sequence that is mutually disjoint, then you can add to that sequence the empty set once or more than once without hurting that property, after all for any subset $A$ it holds that $A \cap \emptyset=\emptyset$.

Secondly in this brief section a quick word about partitions. Fix some universe $\Omega$. Recall that if you think of $\Omega$ as a sheet of paper, then you can create a partition of $\Omega$ by grabbing a pair of scissors and cutting it into the pieces. This is a partition because of the following two properties:

none of the pieces overlap with each other (mutually disjoint), and
if you were to glue them back together (which would admittedly make the whole exercise a bit pointless ;)) you get the original sheet back again.

Definition 1.23 (Partition) In some universe $\Omega$, a sequence of non-empty subsets $A_1, \ldots, A_n$ (or: $A_1, A_2, \ldots$) is called a partition of $\Omega$ if:

they are mutually disjoint (cf. Definition 1.20),
\[\bigcup_{i=1}^n A_i=\Omega \quad \left( \text{or: } \bigcup_{i=1}^\infty A_i=\Omega \right).\]

We can also capture this all in one simple sentence: $A_1, \ldots, A_n$ (or: $A_1, A_2, \ldots$) is a partition of $\Omega$ if and only if each $x \in \Omega$ is an element of exactly one of the $A_i$’s.

Sometimes we explicitly specify that the partition is finite (the case $A_1, \ldots, A_n$) or infinite (the case $A_1, A_2, \ldots$). Note that also in the infinite case, as is clear from the notation “$A_1, A_2, \ldots$”, we only work with a countable collection of subsets (cf. Subsection 1.3.2). We do not allow for uncountable collections to be valid partitions.

Example 1.24 (Some examples of partitions) We see an interesting and consequential difference appear here between a countable universe $\Omega$ on the one hand and an uncountable universe $\Omega$ on the other hand (recall from Subsection 1.3.2)! The simplest way to (attempt to) create a partition is to simply put all elements of $\Omega$ in their own set. However:

if $\Omega$ is countable then this works fine. For instance, if $\Omega=\N$ then that procedure yields the collection/sequence of subsets $\{0\}, \{1\}, \ldots$ and this indeed satisfies the criteria from Definition 1.23 to be a partition as you can readily check.
if $\Omega$ is uncountable on the other hand, for instance if $\Omega=\R$, then this procedure yields an uncountable collection of subsets and this is not a valid partition according to Definition 1.23!

Of course there are typically plenty of other, valid choices for partitions possible. For instance in the case $\Omega=\R$, consider the collection consisting of the intervals $A_n=(n,n+1]$ for all $n \in \Z$. This is a countable collection because the “index set” $\Z$ is countable (cf. Definition 1.16, also recall the discussion in Remark 1.18 for instance), these intervals are mutually disjoint, and their union is the whole of $\R$, so this indeed checks all the boxes of Definition 1.23. If you find this argument a bit too loose for comfort and would prefer to align more stricty with the notation used in Definition 1.23, then just organise listing these intervals as follows for instance: $A_1=(0,1]$, $A_2=(-1,0]$, $A_3=(1,2]$, $A_4=(-2,-1]$ etc.

1.3.4 Sets of sets

This is the final stop one in our (somewhat lengthy, sorry!) journey reminding you of some useful things that you have seen before. In preparation for our discussion about $\sigma$-algebras in Section 1.5, let’s talk a little bit more about “sets of sets”. We’ve actually seen them several times before in the previous sections already, for instance in Subsection 1.3.3, but it’s probably good to nail them on the head a bit!

Let $\Omega$ be any universe. Of course we can create subsets in $\Omega$: choose one or more elements from $\Omega$ and put these together in a bag that is our subset. Let’s say that we create three subsets: $A_1, A_2, A_3$. Now imagine that we take these three subsets/bags, and put these together in a new bag. This new bag then holds three items. These items are not elements from $\Omega$, but they are subsets in $\Omega$. We have then created, what we like to call, a collection of subsets (of $\Omega$).

Recall that mathematically speaking, as also discussed at the start of Subsection 1.3.1, a set is nothing but a bag holding items, of whatever type, that we call the set’s elements. So the bag/collection that we have created in the previous paragraph is a set itself, a “set of sets” if you will! For clarity of communication and notation, we will always call such a guy a collection (of sets) and for its name we will always use a calligraphic font: $\mathcal{A}$, $\mathcal{B}$, $\mathcal{C}$, … Cf. Figure 1.1 for a visualisation.

Definition 1.25 (Collection of subsets) Let $\Omega$ be any universe. A collection of subsets (of $\Omega$) is a set $\mathcal{C}$ whose elements are subsets of $\Omega$.

We also allow for $\mathcal{C}=\emptyset$, which is the collection of subsets from $\Omega$ which we have simply chosen to fill with no subsets from $\Omega$ at all, an empty bag!

Example 1.26 (Some collections of subsets) Take $\Omega=\N$ and consider the following collections: \[\begin{align*} \mathcal{A}=\{\{1,2\},\{2,3\}\}, \quad \mathcal{B} &= \{\emptyset,\Omega\}, \\ \mathcal{C}= \{\{2\}\}, \quad \mathcal{D}&=\{\{5,10\},2,\{3,4,5,...\}\}. \end{align*} \]

Then $\mathcal{A}$ is a set/collection holding two elements, namely $\{1,2\}$ and $\{2,3\}$. These are both subsets of $\Omega$, so $\mathcal{A}$ is a valid collection of subsets as defined in Definition 1.25. The same holds true for $\mathcal{B}$: its elements are $\emptyset$ and $\Omega$, both subsets of $\Omega$ (if you’re in doubt about $\emptyset$, recall Definition 1.8) so all good. Also $\mathcal{C}$ is still fine: it contains only one element, the singleton $\{2\}$, which is indeed a subset of $\Omega$. However $\mathcal{D}$ is not a valid collection of subsets: one of its elements is $2$, which is not a subset of $\Omega$ (but an element of it)!

In Definition 1.25 we already saw the smallest possible collection of subsets of $\Omega$, namely the one containing no elements (subsets of $\Omega$) at all: $\mathcal{C}=\emptyset$. At the other extreme, the largest possible collection is the one that simply contains all possible subsets of $\Omega$, which has a special name:

Definition 1.27 (The powerset) Let $\Omega$ be any universe. The powerset of $\Omega$, denoted by $\mathcal{P}(\Omega)$, is the collection of subsets (cf. Definition 1.25) that contains all possible subsets of $\Omega$ as its elements.

Note that no matter what $\Omega$ is, $\mathcal{P}(\Omega)$ will contain at least the two elements $\emptyset$ and $\Omega$.

Example 1.28 (Simple examples of powersets) Consider any universe $\Omega$ consisting of two elements (whatever they are), say $\Omega=\{\omega_1,\omega_2\}$. Then if you think about it for a moment, you’ll agree that all possible subsets of $\Omega$ are \[\emptyset, \ \{\omega_1\}, \ \{\omega_2\}, \ \{\omega_1,\omega_2\}=\Omega.\] Hence its powerset (cf. Definition 1.27) is the collection \[\mathcal{P}(\Omega)=\big\{ \emptyset, \ \{\omega_1\}, \ \{\omega_2\}, \ \{\omega_1,\omega_2\} \big\}.\]

Let’s do another one, now some $\Omega$ containing three elements, say $\Omega=\{\omega_1,\omega_2,\omega_3\}$. Again you may have to think about it a little bit but then you’ll agree that all possible subsets of this $\Omega$ are \[\emptyset, \ \{\omega_1\}, \ \{\omega_2\}, \ \{\omega_3\},\] \[\{\omega_1,\omega_2\}, \ \{\omega_1,\omega_3\}, \ \{\omega_2,\omega_3\},\] \[\{\omega_1,\omega_2,\omega_3\}=\Omega\] and hence its powerset is \[\mathcal{P}(\Omega)=\big\{ \emptyset, \ \{\omega_1\}, \ \{\omega_2\}, \ \{\omega_3\}, \ \{\omega_1,\omega_2\}, \ \{\omega_1,\omega_3\}, \ \{\omega_2,\omega_3\}, \ \{\omega_1,\omega_2,\omega_3\} \big\}.\]

More generally, if $\Omega$ contains $n$ elements then it has $2^n$ possible subsets (and hence its powerset has $2^n$ elements). If $\Omega$ has infinitely many elements though, no matter whether countable or uncountable, then its powerset is uncountable! Cf. Exercise 1.3.

Ok Kees, right, so, did we really need to have a whole dedicated section to talk about this stuff?? Well you know, here is the thing that makes the brain of you poor guys sometimes suddenly short circuit and is hence worth to stress quite hard: the fact these collections of subsets are, at the end of the day, sets themselves, means that all the usual set relationships (cf. Definition 1.8) and set operations (cf. Definition 1.10) can be applied to such collections of subsets! These relationships and operations (of course) work exactly like you would expect, as long as you keep the one golden rule in mind: when working with collections of subsets, their elements are always subsets of $\Omega$, never elements of $\Omega$ itself! Or, equivalently: rather than working with the universe $\Omega$ to pick elements from, we now work with $\mathcal{P}(\Omega)$ to pick elements from.

This is a case where an example is worth a thousand words:

Example 1.29 (Applying set relationships and operations to collections of subsets) Consider again the case where $\Omega$ has two elements, say $\Omega=\{\omega_1,\omega_2\}$, with (cf. Example 1.28) \[\mathcal{P}(\Omega)=\big\{ \emptyset, \ \{\omega_1\}, \ \{\omega_2\}, \ \Omega \big\},\] with some possible collections of subsets of $\Omega$:

$\mathcal{A}=\emptyset$ (the “smallest” one, empty i.e. not containing any elements from $\mathcal{P}(\Omega)$)
$\mathcal{B}=\big\{ \emptyset, \ \{\omega_2\} \big\}$
$\mathcal{C}=\big\{ \{\omega_1\} \big\}$
$\mathcal{D}=\big\{ \{\omega_2\}, \ \Omega \big\}$
$\mathcal{E}=\big\{ \emptyset, \ \{\omega_1\}, \ \{\omega_2\}, \ \Omega \big\}$ (the “largest” one containing all possible elements i.e. $\mathcal{P}(\Omega)$ itself).

Set relationships: the usual set relationships from Definition 1.8 apply in the obvious way, as long as you realise that when we’re talking about “elements” we now mean “subsets of $\Omega$”. For instance:

The subset $\{\omega_2\}$ is an element of $\mathcal{C}$ but not an element of $\mathcal{D}$, i.e. $\{\omega_2\} \in \mathcal{C}$ and $\{\omega_2\} \not\in \mathcal{D}$.
Tickling your brain a bit: it holds that $\emptyset \in \mathcal{B}$ but $\emptyset \not\in \mathcal{A}$!
It holds that $\mathcal{C} \subseteq \mathcal{D}$, because each elements of $\mathcal{C}$ (only one in this case, namely $\{\omega_1\}$) is also an element of $\mathcal{D}$.

Etc.

Set operations: the usual set operations from Definition 1.10 also apply in the obvious way, as long as you still keep in mind that when we’re talking about “elements” we now mean “subsets of $\Omega$”. To quickly run through them, where we let $\mathcal{F}$ and $\mathcal{G}$ denote any two collections of subsets of $\Omega$.

Complement: $\mathcal{F}^c$ is the collection of all subsets of $\Omega$ that are not an element of $\mathcal{F}$ i.e. \[\mathcal{F}^c=\{ A \in \mathcal{P}(\Omega) \, | \, A \not\in \mathcal{F} \}.\] For example: \[\mathcal{D}^c = \big\{ \emptyset, \ \{\omega_1\} \big\} \quad \text{and} \quad \mathcal{A}^c=\mathcal{P}(\Omega).\]
Union: $\mathcal{F} \cup \mathcal{G}$ is the collection of all subsets of $\Omega$ that are either an element of $\mathcal{F}$ or an element of $\mathcal{G}$ (or both) i.e. \[\mathcal{F} \cup \mathcal{G}=\{ A \in \mathcal{P}(\Omega) \, | \, A \in \mathcal{F} \text{ or } A \in \mathcal{G} \}.\] For example: \[\mathcal{B} \cup \mathcal{C}= \big\{ \emptyset, \ \{\omega_1\}, \ \{\omega_2\} \big\}.\]
Intersection: $\mathcal{F} \cap \mathcal{G}$ is the collection of all subsets of $\Omega$ that are both an element of $\mathcal{F}$ as well as an element of $\mathcal{G}$ i.e. \[\mathcal{F} \cap \mathcal{G}=\{ A \in \mathcal{P}(\Omega) \, | \, A \in \mathcal{F} \text{ and } A \in \mathcal{G} \}.\] For example: \[\mathcal{B} \cap \mathcal{D} = \big\{ \{\omega_2 \} \big\} \quad \text{and} \quad \mathcal{B} \cap \mathcal{C} = \emptyset.\]
Set difference: $\mathcal{F} \setminus \mathcal{G}$ is the collection of all subsets of $\Omega$ that are an element of $\mathcal{F}$ but not an element of $\mathcal{G}$ i.e. \[\mathcal{F} \setminus \mathcal{G}=\{ A \in \mathcal{P}(\Omega) \, | \, A \in \mathcal{F} \text{ and } A \not\in \mathcal{G} \}.\] For example: \[\mathcal{D} \setminus \mathcal{B}=\big\{ \Omega \big\}.\]

Figure 1.1: A visualisation of a universe $\Omega$ with two collections of subsets, say $\mathcal{A}$ (containing three elements, in red) and $\mathcal{B}$ (containing five elements, in blue)

Exercises

You can now do Exercise 1.3.

1.4 What’s next?

Let’s take stock for a moment and talk about the overall big picture of what we’re trying to achieve! In the above Section 1.2 and Section 1.3 we’ve done a review of some proving logic and a lot of stuff to do with sets. But why, and what’s next? As outlined in Section 1.1, we’re gradually building up the measure theoretic foundations of modern Probability Theory in our first four chapters, and then in later chapters build some impressive buildings on top of that foundation: stochastic processes, martingales etc.

Recall from your earlier Prob courses that in Probability Theory, everything starts with an experiment whose possible outcomes are collected in an outcome space or sample space, often denoted by $\Omega$. Events are subsets of $\Omega$ which may or may not happen (i.e. when we do the experiment, an event/subset happens if and only if the observed outcome of the experiment is an element of that event/subset), and each event has a probability of happening. On top of that basic structure we can then introduce random variables i.e. mappings from $\Omega$ to $\R$ that measure something of relevance about the possible outcomes of our experiment, and ultimately stochastic processes and martingales which are at their core nothing but collections of such random variables.

You’d be forgiven for not having this structure entirely sharp anymore, because we are a bit lazy (with justification!) and as you got a bit further into Probability Theory, examples/questions often were as short as “Let $X$ be random variable with [this-or-that] distribution. Compute [one-thing-or-the-other].”. No mentioning of all most of that structure from the previous paragraph at all! However, it is always there, in any Probability context, even if it’s not (explicitly) specified. The justification for this laziness is twofold:

it can be shown that you can always construct an experiment, outcome space, events, probabilities etc. to support a random variable with any distribution that you want (we’ll see this later);
in such an example/question that focusses on the “behaviour” of a random variable it doesn’t make any difference which experiment, outcome space, events, probabilities etc. you exactly choose (we’ll see this later as well!).

So this is a bit like planning a holiday and only talking about the activities you want to do while out there, then have one of your friends interject with “But don’t we first of all need a hotel to stay in?”, to which another friend, the lazy one, replies “Yeah but there are plenty of decent hotels out there, we know for sure we can pick one once we get there and it makes no difference which one exactly, so we don’t need to actually discuss that.”.

Going forward, we do need to consider that whole structure of experiment, outcome space, events, probabilities etc. carefully and build it up piece by piece, because your basic Probability courses have swept a thing or two under the carpet! That is our key goal in the first four chapters.

Definition 1.30 (A (probabilistic) experiment) When we are working within Probability Theory, we always (no matter whether it’s explicitly mentioned or not!) do that in the context of a (probabilistic) experiment i.e. some (“real world” or imagined) process that generates an outcome that (normally) cannot be predicted. Mathematically, this involes the following ingredients:

the outcome space or sample space: the set containing all possible outcomes of the experiment, typically denoted by $\Omega$;
events are subsets of $\Omega$, each of which either happens (contains the observed outcome) or doesn’t happen (doesn’t contain the observed outcome) when the experiment is done;
each event has a probability of happening, a number from the interval $[0,1]$ expressing how likely it is that the event in question happens if the experiment is done.

Later on, we will add to this structure random variables (in Chapter 2) and stochastic processes (in Chapter 5).

A large part of this structure is intimately shared with the field measure theory (hence also the phrasing “measure theoretic foundations of modern Probablity Theory”!). Therefore in these first four chapters (in particular the first three), we will discuss things not only with a Probability hat on but slightly more generally. Of course, for all the important concepts we will see we will always points out what their role in Probablity Theory is.

There is one bit of useful but ad hoc (i.e. I made it up!) terminology to introduce. We’ve seen in the above Section 1.3 at several moments that there are some fundamental differences between dealing with countable sets on the one hand and uncountable sets on the other hand (cf. Subsection 1.3.2 in particular). This will have some technical yet quite serious ramifications for our discussions of experiments as well!

Definition 1.31 (Simple vs advanced experiment (ad hoc terminology!)) Let $\Omega$ be the outcome space of some experiment as in Definition 1.30 i.e. the set containing all possible outcomes. Then we call the experiment (cf. Definition 1.16 and Example 1.17):

simple if $\Omega$ is countable,
advanced if $\Omega$ is uncountable.

So in a simple experiment, we can list and/or “walk through” all possible outcomes or, equivalently, we can express $\Omega$ as (using generic notation for its elements) $\Omega=\{\omega_1,\ldots,\omega_n\}$ (finite) or $\Omega=\{\omega_1,\omega_2,\ldots\}$ (countably infinite). For instance, if we roll a dice so that $\Omega=\{⚀, ⚁, ⚂, ⚃, ⚄, ⚅ \}$ or if we shoot a rocket in a random direction into space and count how many stars it will encounter so that $\Omega=\{0,1,2,\ldots\}$.

In an advanced experiment, it is not possible to list and/or “walk through” all possible outcomes. For example, suppose that you go out on a random day in Manchester and measure (with infinite precision) how many liters of rain falls during the day in your garden, so that $\Omega=[0,\infty)$.

Remark 1.32. In the example about rain in Definition 1.31, note that “with infinite precision” is a crucial addition — we mean by this that we make our measurement with infinitely many decimals of precision. Indeed, if we were to use a precision of say two decimals only, like 2.56 liter or whatever, then this is a rational number and therefore the set of all possible outcomes $\Omega$ would be a subset of $\Q$ and hence countable (cf. Example 1.17 iv).

It does make you feel a bit uneasy though, doesn’t it, measuring something with “infinite precision”. My dad has a very fancy rain gauge but he would be very jalous of a device being able to do that! It’s hard to imagine that this is realistically possible. However, that’s not really the point. This comes back to what we highlighted in Section 1.1 already as well: an advanced experiment/uncountable outcome space may (arguably) not be very realistic, mathematically it is a perfectly valid construction and including them in our thinking gives rise to a whole suite of results and techniques that make modern Probability Theory so beautiful, powerful and impressive!

In the remainder of this chapter, we will take a close look at the elements that appear in Definition 1.30:

Challenge 1: Outcome spaces and events in Section 1.5. But Kees, you know, the outcome space is just some set, and events are just subsets of that set. Is there really anything more to be said here? As it turns out: yes there is! We’ll discuss how in advanced experiments (cf. Definition 1.31), for deep technical reasons, not all subsets can be considered “valid” events, and we’ll see that the natural solution is to make use of $\sigma$-algebras.
Challenge 2: Probabilities of events, in the form of (probability) measures, in Section 1.6.

1.5 Sigma-algebras & measurable spaces

So let’s take up the first challenge we formulated in Section 1.4! Consider some probabilistic experiment with outcome space $\Omega$ (cf. Definition 1.30). As we already said, and know very well from earlier Prob courses, an event is nothing but a subset of (the universe) $\Omega$. What about the other way around, is any subset of $\Omega$ also an event? Well yeah, why not, what are you going on about Kees? Here’s the thing: surprisingly, it turns out that the maths forces us to answer this question with no, not always.

As long as we are working with simple experiments (cf. Definition 1.31) life is simple: we can simply consider every subset of $\Omega$ to be an event. Or, equivalently: the collection of all possible events is(/can be taken to be) simply the collection of all possible subsets of $\Omega$ i.e. the powerset $\mathcal{P}(\Omega)$ (cf. Definition 1.27).

However here is a price to pay for our flirtations with the Dark Curse of Uncountability: by desperately wanting to include advanced experiments (cf. Definition 1.31) as well life gets more complicated. And this was found out the hard way! In the development of this theory, initially some of our ancestor mathematicians (smart people!) did make the natural assumption that also in this case, every possible subset of $\Omega$ can be considered to be an event. But if you then go on and try to define probabilities for all these events, you run into problems: weird, counterintuitive, unacceptable properties start cropping up. You’ll have to take my word for this for now, as we don’t yet have the tools to see this in action, but we will see this unfold in Subsection 1.6.1!

The solution to avoid these problems: take a smaller (than the powerset) collection of subsets of $\Omega$ and consider only these subsets to be “valid” events. The subsets not in this collection we simply ignore: we call them unmeasurable and do not attempt to assign them a probability at all. Such a collection of “valid” events should have a certain structure (to make it suitable for working with probabilities), and that’s where $\sigma$-algebras come in, the topic of this section.

So that’s the motivation (from a probabilistic point of view), now let’s talk in more detail about these $\sigma$-algebras. For the remander of this section, recall that we discussed collections of subsets in general in Subsection 1.3.4, and that we use the terminology “universe” to indicate a “master” set in which all the action happens.

Definition 1.33 (Sigma-algebras and measurable spaces) Consider some universe $\Omega$ with powerset $\mathcal{P}(\Omega)$ (cf. Definition 1.27). A collection of subsets of $\Omega$ denoted by $\mathcal{F}$ (so, $\mathcal{F} \subseteq \mathcal{P}(\Omega)$) is called a $\sigma$-algebra (on $\Omega$) if it satisfies the following conditions:

$\emptyset \in \mathcal{F}$,
if $A \in \mathcal{F}$ then also $A^c \in \mathcal{F}$,
if $A_1, A_2, \ldots \in \mathcal{F}$ then also \[\bigcup_{i=1}^\infty A_i \in \mathcal{F}.\]

Conditions ii and iii are commonly put in words as, respectively, “$\mathcal{F}$ is closed under complements” and “$\mathcal{F}$ is closed under countable unions”.

We call a pair $(\Omega,\mathcal{F})$ of a universe and a $\sigma$-algebra a measurable space. Further we call the elements of $\mathcal{F}$ measurable sets (or also $\mathcal{F}$-measurable sets if we want to stress the $\sigma$-algebra we’re working with).

Some notes:

Observe that if condition iii is satisfied, then the analogue holds for a finite sequence of elements of $\mathcal{F}$: if $A_1, \ldots, A_n \in \mathcal{F}$ then also \[\bigcup_{i=1}^n A_i \in \mathcal{F}.\] The proof is simple: extend the sequence to an infinite one, by setting $A_{n+1}:=\emptyset$, $A_{n+2}:=\emptyset$, … and then apply condition iii above to that infinite sequence.
Note that condition iii above only applies to any countable collection of elements of $\mathcal{F}$ (cf. Remark 1.18), if you were to take an uncountable collection of elements of $\mathcal{F}$ (provided that $\mathcal{F}$ itself is uncountable so that this is possible) then their union (which is well defined, recall from Remark 1.18) may or may not be an element of $\mathcal{F}$!
As discussed above, our primary motivation (but not only use, as we will see) for introducing $\sigma$-algebras is that in the probability context (where $\Omega$ has a specific interpretation, namely the outcome space of an experiment, cf. Definition 1.30) they define suitable smaller (than the powerset) collections of subsets to consider as our (“valid”) events that we can assign a probability of happening.

From this perspective, all three conditions are also very natural when you relate them back to how we like to be able to reason about events in Probability Theory. If $A$ is an event, then we naturally also want to be able to talk about “$A$ does not happen” i.e. $A^c$ should be an event (condition ii). Similarly if $A$ and $B$ are events, then the same holds for “$A$ or $B$ happens” i.e. $A \cup B$ should also be an event (condition iii). You may wonder about “and” events i.e. intersections, but it turns out that we get these for free (cf. Proposition 1.35 below). The empty set, the only event that never happens, finally (condition i) also has a natural place here: for instance if $A$ and $B$ are disjoint events, then “$A$ and $B$ happens” is a statement about $A \cap B=\emptyset$ and hence $\emptyset$ should also be an event.

Example 1.34 (Simple examples of sigma-algebras) On any given (not entirely trivial) universe many $\sigma$-algebras exist. Here are some examples of simple ones. On any universe $\Omega$, the following collections of subsets $\mathcal{F}$ are all $\sigma$-algebras:

$\mathcal{F}=\mathcal{P}(\Omega)$ (the powerset itself is trivially also a $\sigma$-algebra and it is clearly the largest possible one),
$\mathcal{F}=\big\{ \emptyset, \Omega \big\}$ (the smallest possible one),
for any subset $B \subseteq \Omega$ fixed, $\mathcal{F}=\big\{ \emptyset, B, B^c, \Omega \big\}$.

Example i is obvious: conditions ii and iii in Definition 1.33 are readily checked because the complement resp. countable union of subsets of $\Omega$ yields again a subset of $\Omega$ and therefore an element of $\mathcal{F}=\mathcal{P}(\Omega)$.

Example ii is clear as well, since $\emptyset^c=\Omega$ and $\Omega^c=\emptyset$, and for condition iii in Definition 1.33, observe that any sequence $A_1, A_2, \ldots \in \mathcal{F}$ we can choose either consists of empty sets only (nobody said that it was required for $A_1, A_2, \ldots$ to be different sets!) so that the union is also the empty set, or otherwise at least one of the $A_i$’s equals $\Omega$ and then the union is $\Omega$.

Example iii can also readily be checked in the same way, have a look at that one yourself!

The conditions set out in Definition 1.33 are appropriately “minimal”⁴, we get (almost) for free that a $\sigma$-algebra is also closed under countable unions and set differences:

⁴ As you generally want in maths: you want the conditions that define some concept $X$ as “minimal” as possible (to make proving “blahblah is an $X$” as straightforward/little work as possible), while on the other hand you want to derive the “maximal” number of properties $X$ has i.e. the more the better, as that helps you understand the concept and is useful in later proofs to make steps like “since (something) is an $X$, it follows from property (blah) that …”

Proposition 1.35 (Some properties of sigma-algebras) Let $\mathcal{F}$ be a $\sigma$-algebra on a universe $\Omega$ (cf. Definition 1.33). Then:

if $A_1, A_2, \ldots \in \mathcal{F}$ then also \[\bigcap_{i=1}^\infty A_i \in \mathcal{F}\] (“closed under countable intersections”),
if $A,B \in \mathcal{F}$ then also $A \setminus B \in \mathcal{F}$ (“closed under set differences”).

Some notes:

Analogue to what we saw in Definition 1.33: property i also holds for a finite sequence $A_1, \ldots, A_n \in \mathcal{F}$, indeed just extend it to an infinite sequence by setting $A_{n+1}:=\Omega$, $A_{n+2}:=\Omega$, … and apply property i above to this infinite sequence. For this, note that $\Omega$ is indeed guaranteed to be an element of $\mathcal{F}$, because $\emptyset$ is (by Definition 1.33 i) and then use Definition 1.33 ii.

Proof. For property i, take any $A_1, A_2, \ldots \in \mathcal{F}$. The key idea is this: we know from Definition 1.33 that $\mathcal{F}$ is closed under (countable) unions and complements. So ideally we would like to rewrite the intersection of the $A_i$’s in some form using only unions and complements so that we can use these facts. Do we know of a way to do that? Well yes we do, the good old De Morgan’s laws (cf. Definition 1.10)! It tells us that \[\left( \bigcap_{i=1}^\infty A_i \right)^c = \bigcup_{i=1}^\infty A_i^c\] and if we now take the complement on both sides then we see our desired intersection in the left hand side: \[\bigcap_{i=1}^\infty A_i = \left( \bigcup_{i=1}^\infty A_i^c \right)^c. \tag{1.8}\] So what we have now left to do is to show that the right hand side of Equation 1.8 is an element of $\mathcal{F}$. For this, we work “inside out”. First observe that for any $i=1,2,\ldots$, we have assumed that $A_i \in \mathcal{F}$ and hence also $A_i^c \in \mathcal{F}$ (“closed under complements”). Using “closed under countable unions” then also \[\bigcup_{i=1}^\infty A_i^c \in \mathcal{F}.\] And finally, again using “closed under complements” it then follows that also the right hand side of Equation 1.8 is an element of $\mathcal{F}$, as we needed.

For property ii, take any $A, B \in \mathcal{F}$. We use the same idea as we did for property i above: can we somehow express the set difference $A \setminus B$ in terms of (countable) unions, complements, and intersections (after all, we have now proven property i)? Yes we can! As you’ll probably remember well and otherwise it’s not hard to guess from the def (cf. Definition 1.10): \[A \setminus B=A \cap B^c\] (it’s also straightforward to prove). So we need to show that the right hand side is an element of $\mathcal{F}$, which is not very hard — again working “inside out”: we assumed that $B \in \mathcal{F}$, so also $B^c \in \mathcal{F}$ (“closed under complements”), and then also $A \cap B^c \in \mathcal{F}$ (since we assumed that $A \in \mathcal{F}$ and “closed under intersections” from property i above).

Ok! So let’s take a moment to see where we’re at: we have motivated why we need $\sigma$-algebras, we have defined them, derived some properties and have seen some simple examples. There’s three more things we would like to do with them in this section:

The examples we saw in Example 1.34 were quite basic and simple. There is a good reason for this: in general, it is quite difficult to explicitly write down more complicated examples. That’s a problem, because how do we then get a good handle on any particular not-so-easy $\sigma$-algebra to do some work with⁵? We’ll discuss a twofold solution: we can indirectly/implicitly specify general $\sigma$-algebras using a generation procedure (cf. Subsection 1.5.1), and we will see that the special category of finite $\sigma$-algebras do have some very neat structure that give us a full picture (cf. Subsection 1.5.2).
There is one very prominent example, especially in Probability Theory: the Borel $\sigma$-algebra on the universe $\Omega=\R$. We’ll introduce this new friend in Subsection 1.5.3.
Finally, we would like to discuss a very interesting interpretation of a $\sigma$-algebra as representing a certain “level of information” about the outcome of an experiment in the Probability Theory context, which later feeds into our discussion of conditional expectations in Chapter 4. We will do this in Subsection 1.5.4.

⁵ There is a more delicate approach to $\sigma$-algebras, via so-called pi-systems and Dynkin systems, which provide a bit more structural insight, but our time is limited and therefore we don’t go down this road

Exercises

You can now do Exercise 1.4–Exercise 1.6.

1.5.1 Generating sigma-algebras

As mentioned above, it is not so easy to explicitly write down more complicated examples of $\sigma$-algebras (cf. Definition 1.33) than the more trivial ones we have seen in Example 1.34. To find a way forward with this, we essentially only need one observation, as follows: if $\mathcal{F}$ and $\mathcal{G}$ are both $\sigma$-algebras on some universe $\Omega$, then their intersection \[\mathcal{F} \cap \mathcal{G}\] (i.e. the collection consisting of all subsets of $\Omega$ that are elements of both $\mathcal{F}$ and $\mathcal{G}$) is also a $\sigma$-algebra on $\Omega$. This is not too difficult to prove, give it a go! And in fact this observation holds in a lot more generality: if you take any (countable or uncountable!) collection of $\sigma$-algebras on $\Omega$, then their intersection is again a $\sigma$-algebra on $\Omega$.

Here is what this observation allows us to. Take any collection of subsets, say $\mathcal{C}$, of some universe $\Omega$ (so $\mathcal{C}$ itself may or may not be a $\sigma$-algebra). Consider any $\sigma$-algebra $\mathcal{F}$ on $\Omega$ with the property that $\mathcal{C} \subseteq \mathcal{F}$ (i.e. any subset of $\Omega$ that is an element of $\mathcal{C}$ is also an element of $\mathcal{F}$). Note that there is at least one such $\sigma$-algebra, namely the powerset $\mathcal{P}(\Omega)$ (cf. Definition 1.27), but there may be many more. Now define $\mathcal{G}$ to be the intersection of all these $\sigma$-algebras. Then this guy has the following properties:

$\mathcal{G}$ is a $\sigma$-algebra (by the observation we made above),
$\mathcal{C} \subseteq \mathcal{G}$ (since any element of $\mathcal{C}$ is also an element of each of the $\sigma$-algebras that appeared in our intersection),
if $\mathcal{H}$ is a(nother) $\sigma$-algebra with the property that $\mathcal{C} \subseteq \mathcal{H}$, then $\mathcal{G} \subseteq \mathcal{H}$ (after all, this $\mathcal{H}$ must be one of the $\sigma$-algebras we put in the intersection mix to create $\mathcal{G}$, so $\mathcal{G}$ cannot contain any elements that are not also an element of $\mathcal{H}$).

We can capture these properties nicely in one sentence: “$\mathcal{G}$ is the smallest $\sigma$-algebra that satisfies $\mathcal{C} \subseteq \mathcal{G}$ i.e. that contains all the elements of $\mathcal{C}$”. We call $\mathcal{G}$ the $\sigma$-algebra generated by $\mathcal{C}$, and the symbol we use for it is $\sigma(\mathcal{C})$. This procedure opens up the following strategy for identifying a suitable more complex $\sigma$-algebra for whatever the purpose at hand is: write down all the subsets of $\Omega$ that you definitely want to be in your $\sigma$-algebra, put these together in a collection $\mathcal{C}$, and then work with the generated $\sigma$-algebra $\sigma(\mathcal{C})$!

Sure, there is some lack of precise control here: the end result is a $\sigma$-algebra which contains all the elements of $\mathcal{C}$, but (unless $\mathcal{C}$ itself is already a $\sigma$-algebra!) there will also be other elements in $\sigma(\mathcal{C})$, the “padding” that is needed to make it a $\sigma$-algebra if you will, and in general we have no idea what these elements exactly are. We’ll just have to live with that though!

Definition 1.36 (Generating sigma-algebras) Let $\mathcal{C}$ be any collection of subsets from a universe $\Omega$. Then there exists a smallest $\sigma$-algebra (cf. Definition 1.33) containing all elements of $\mathcal{C}$. We call this the $\sigma$-algebra generated by $\mathcal{C}$ and denote it by $\sigma(\mathcal{C})$. By “smallest” we mean that if $\mathcal{H}$ is some other $\sigma$-algebra containing all elements of $\mathcal{C}$, then $\sigma(\mathcal{C}) \subseteq \mathcal{H}$.

Example 1.37 (Simple example of generating a sigma-algebra) We’ll see more interesting examples shortly, but just to create a little bit of feel: let $\Omega$ be any universe and fix some subset $B \subseteq \Omega$. Consider the collection consisting of just one element: $\mathcal{C}=\{B\}$. Can we find out what $\sigma(\mathcal{C})$ as defined in Definition 1.36 above is?

Well, since $\sigma(\mathcal{C})$ is a $\sigma$-algebra, it must contain $\emptyset$ and $\Omega$ as elements (cf. Definition 1.33 i and ii). Further it must contain all the elements of $\mathcal{C}$, so in particular it must contain $B$ as an element. Again applying Definition 1.33 ii, it must then also contain $B^c$ as an element. Putting all these toghether, we know so far that \[\{\emptyset,B,B^c,\Omega\} \subseteq \sigma(\mathcal{C}).\] On the other hand, in Example 1.34 we have already seen that $\{\emptyset,B,B^c,\Omega\}$ is a $\sigma$-algebra. And clearly $\{\emptyset,B,B^c,\Omega\}$ contains all the elements of $\mathcal{C}$ (which is just $B$ eh). So it also follows from Definition 1.36 that \[\sigma(\mathcal{C}) \subseteq \{\emptyset,B,B^c,\Omega\}.\] Lovely! Hence we can conclude that (recall Remark 1.9) \[\sigma(\mathcal{C}) = \{\emptyset,B,B^c,\Omega\}.\]

Exercises

You can now do Exercise 1.7–Exercise 1.8.

1.5.2 Special case: finite sigma-algebras

This subsection is basically coming from the same sentiment that the previous section on generating $\sigma$-algebras was rooted in: damn a general $\sigma$-algebra is complicated! Where in the above subsection we talked about how we can (implicitly) construct more complex $\sigma$-algebras, in this subsection we discuss how there is a particular category of $\sigma$-algebras, namely the finite ones (i.e. the ones that have finitely many elements) that actually do have a beautiful internal structure that make them easier to understand and study.

To start this story, let $\Omega$ be any universe and let $A_1, \ldots, A_n$ be a finite partition (cf. Definition 1.23) of $\Omega$. Recall the nice picture: imagine that $\Omega$ is a sheet of paper, take a pair of scissors, and cut $\Omega$ up in $n$ pieces which are then your $A_1, \ldots, A_n$.

We claim that the $\sigma$-algebra generated by the collection $\mathcal{C}=\{A_1,\ldots, A_n\}$ (cf. Definition 1.36) consists exactly of the following subsets of $\Omega$:

$\emptyset$,
$A_1, \ldots, A_n$ themselves,
the union of any two elements from $\mathcal{C}$ i.e. $A_i \cup A_j$ for any $i,j \in \{1,\ldots,n\}$,
the union of any three elements from $\mathcal{C}$ i.e. $A_i \cup A_j \cup A_k$ for any $i,j,k \in \{1,\ldots,n\}$,
etc.
the union of any $n$ elements from $\mathcal{C}$, of which there is actually only one, namely $A_1 \cup A_2 \cup \ldots \cup A_n=\Omega$ (recall that we’re working with a partition!).

In one sentence: take the empty set, add the elements of $\mathcal{C}$, and add any possible union of any number of elements from $\mathcal{C}$. Or, in terms of your pieces of paper: the empty set plus any possible selection of any number of your pieces of paper! To maybe illustrate this a little bit further, the total number of (distinct) subsets of $\Omega$ we have listed above is (each term corresponding to one of the bullet points above, in order): \[1+n+\binom{n}{2}+\binom{n}{3}+ \ldots + \binom{n}{n}=2^n\] (using the good old binomial theorem :)). There is a more direct argument to get to $2^n$ as well if you’d like to have something to think about!

We prove our claim in two steps. The first step is to prove that the collection listed in the above bullet points, let’s call it $\mathcal{F}$, is indeed a (finite) $\sigma$-algebra (cf. Definition 1.33). Not at all difficult, just a bit annoying to write down:

clearly $\emptyset \in \mathcal{F}$;
if you take any element from $\mathcal{F}$, say the union of $m$ of elements from $\mathcal{C}$ (where $m=1$ we understand as just an element from $\mathcal{C}$), then since $\mathcal{C}$ is a partition, its complement is the union of the “other” $n-m$ elements of $\mathcal{C}$ and hence indeed also an element of $\mathcal{F}$;
because taking unions is associative, if you take the union of any number of elements from $\mathcal{F}$ you end up with a union of unions of elements from $\mathcal{C}$, which can always be simplified to “just” a union of elements from $\mathcal{C}$ and hence an element from $\mathcal{F}$!

For the second step, looking at Definition 1.36, it remains to show that $\mathcal{F}$ is the smallest $\sigma$-algebra containing all the elements of $\mathcal{C}$. An argument for this is that if you leave out any of the elements of $\mathcal{F}$ we listed above to create a smaller collection than $\mathcal{F}$, then this smaller collection can’t possibly be a $\sigma$-algebra — not very hard to see if you think about it for a moment!

So, ok, the above feels a bit more like an annoying puzzle than proper maths maybe, but our reward is the glorious conclusion: we have now discovered a way to fully explicitly construct non-trivial $\sigma$-algebras! And, as cherry on the top of this explicit cake, this whole construction turns out to also work the other way around: any finite $\sigma$-algebra on $\Omega$ is in fact of the above form i.e. generated by a finite partition of $\Omega$!

Proposition 1.38 (Characterisation of finite sigma-algebras) Take any universe $\Omega$. Then we have the following.

If $A_1, \ldots, A_n$ is a finite partition (cf. Definition 1.23) of $\Omega$, then the $\sigma$-algebra (cf. Definition 1.33) generated by $\mathcal{C}=\{A_1,\ldots, A_n\}$ i.e. $\sigma(\mathcal{C})$ (cf. Definition 1.36) is a finite $\sigma$-algebra consisting of the following elements: $\emptyset$, $A_1, \ldots, A_n$, plus any possible union of any number of elements of $\mathcal{C}$.
If $\mathcal{F}$ is a finite $\sigma$-algebra on $\Omega$, then there exists a (unique) finite partition $A_1, \ldots, A_n$ of $\Omega$ so that $\mathcal{F}=\sigma(\{A_1, \ldots, A_n\})$.

If $\mathcal{F}$ is a finite $\sigma$-algebra on $\Omega$ generated by a partition $A_1, \ldots, A_n$, then we also call $A_1, \ldots, A_n$ the atoms of $\mathcal{F}$.

Keep firmly in mind: this beautiful structure only holds for finite $\sigma$-algebras!

Remark 1.39. Note that if $\mathcal{F}$ is a finite $\sigma$-algebra on $\Omega$ with atoms $A_1, \ldots, A_n$, then these $A_1, \ldots, A_n$ are the “smallest” elements of $\mathcal{F}$ (well, besides $\emptyset$). After all, as is hopefully clear from the listing of all elements in Proposition 1.38, any other element of $\mathcal{F}$ is a union of at least two of the $A_1, \ldots, A_n$.

Example 1.40 (Some examples of finite sigma-algebras) Let’s quickly look at two examples — one old and one (somewhat) new!

Take any universe $\Omega$ and let $B \subseteq \Omega$. Then the pair $B, B^c$ clearly makes up a finite partition of $\Omega$ (cf. Definition 1.23). Hence according to Proposition 1.38, $\sigma(\{B, B^c\})$ is a finite $\sigma$-algebra whose elements are $\emptyset$, $B$, $B^c$, plus any possible union of any number of elements from the pair $B, B^c$. This latter bit gives us only one new element, namely $B \cup B^c=\Omega$. Hence we arrive at \[\sigma(\{B, B^c\})=\{\emptyset, B, B^c, \Omega\},\] indeed the friend that we previously already encountered in Example 1.34 and in Example 1.37!
Take any finite universe, say $\Omega=\{\omega_1,\ldots,\omega_n\}$. Then the collection of all possible singletons $\{\omega_1\}$, $\{\omega_2\}$, …, $\{\omega_n\}$ forms a perfectly valid (finite) partition of $\Omega$ (cf. Definition 1.23). What is, according to Proposition 1.38, the $\sigma$-algebra generated by this partition? We claim that this is in fact the largest possible $\sigma$-algebra on $\Omega$, indeed the powerset $\mathcal{P}(\Omega)$ (cf. Definition 1.27)! Why? Well take any element $A \in \mathcal{P}(\Omega)$ (other than the empty set for convenience of phrasing) i.e. any subset $A \subseteq \Omega$. Then $A$ can trivially be expressed as the union of all its elements as singletons i.e. as union of elements of our partition, and hence by Proposition 1.38 $A$ is also an element of the generated $\sigma$-algebra!

Note that this example only works for finite universes, after all, otherwise the collection of singletons is not a finite partition and hence Proposition 1.38 does not apply!

1.5.3 Special case: the Borel sigma-algebra

It won’t surprise you that one universe we are particularly interested in is the good old real line i.e. $\Omega=\R$. Of course there are many possible $\sigma$-algebras you could construct on $\R$ using the techniques we have already seen:

take any collection $\mathcal{C}$ of subsets of $\R$ that you fancy, and consider the $\sigma$-algebra generated by it (cf. Definition 1.36);
take some finite partition of $\R$, say the collection $(-\infty,-1]$, $(-1,0]$, $(0,1]$, $(1,\infty)$, and apply Proposition 1.38 to it.

Note that since $\R$ is infinite (in fact uncountable, cf. Example 1.17), there also exist infinitely large $\sigma$-algebras on this space. To specify such a beast is beyond the reach of Proposition 1.38 and hence going through Definition 1.36 is our only option.

The Borel $\sigma$-algebra is one of the most prominent $\sigma$-algebras on $\Omega=\R$, and it is fundamental to modern Probability Theory (and elsewhere!) as we will see later. It is defined as the $\sigma$-algebra generated by all open subsets of $\R$, and because it so prominent it has its own symbol: $\mathcal{B}(\R)$. That is to say, let $\mathcal{C}$ be the collection of all open subsets of $\R$ (cf. Definition 1.13), and then we define $\mathcal{B}(\R):=\sigma(\mathcal{C})$ (cf. Definition 1.36).

Ok. So what kind of sets are elements of $\mathcal{B}(\R)$? Well obviously all open subsets of $\R$. But then also all closed subsets of $\R$ are in there (because each of these is the complement of an open subset, and a $\sigma$-algebra is closed under complements, recall from Definition 1.33). And any interval is in there as well (open and closed intervals are covered in the previous two sentences, and the other types are looked at in Exercise 1.9). And hence also any union or intersection of countably many subsets of $\R$ that are each either open, closed or an interval is in there (cf. Definition 1.33 iii and Proposition 1.35 i)! Wow. Maybe we are asking the wrong question then, and we should look at it from the other side: are there any subsets of $\R$ that are not elements of $\mathcal{B}(\R)$, or did we accidentally define a $\sigma$-algebra that is just the powerset of $\R$? The answer is that there do exist subsets of $\R$ that are not in $\mathcal{B}(\R)$, but it’s not very easy to construct one — we’ll see an example in Subsection 1.6.1 though!

Sometimes we will also want to consider a subset of $\R$, normally an interval $I$, as our universe $\Omega$. We can define the Borel $\sigma$-algebra on this universe in the analogue way: let $\mathcal{C}$ be the collection of all open subsets of $I$ (which are just all open subsets of $\R$ that are contained in $I$) and define the Borel $\sigma$-algebra on $I$ as $\mathcal{B}(I):=\sigma(\mathcal{C})$ (cf. Definition 1.36). In fact, you can obtain $\mathcal{B}(I)$ by taking all elements in $\mathcal{B}(\R)$ and intersecting each of them with $I$ (this is an application of Exercise 1.6).

Definition 1.41 (The Borel sigma-algebra) On the universe $\Omega=\R$, the Borel $\sigma$-algebra is denoted by $\mathcal{B}(\R)$ and is defined as the $\sigma$-algebra generated by all open subsets of $\R$ (cf. Definition 1.36). It contains any open or closed subset of $\R$, any interval, and any finite or (countably) infinite union or intersection of the aforementioned subsets. Basically, it contains any subset of $\R$ you normally run into. However there do exist subsets of $\R$ that are not elements of $\mathcal{B}(\R)$.

The elements of $\mathcal{B}(\R)$ are called Borel measurable sets or just Borel sets (on $\R$).

Analogue, if $I \subseteq \R$ is an interval, we define the Borel $\sigma$-algebra on the universe $\Omega=I$, $\mathcal{B}(I)$, to be the $\sigma$-algebra generated by all open subsets of $I$. It consists of all sets of the form $I \cap B$ for $B \in \mathcal{B}(\R)$.

Remark 1.42. Just to stress: the Borel $\sigma$-algebra $\mathcal{B}(\R)$ is not finite, in fact, it is even uncountable (cf. Definition 1.16). So none of that very neat structure we saw in Subsection 1.5.2 to describe the whole $\sigma$-algebra applies here sadly!

Here is just one example of how much more subtle things work here — the Dark Curse of Uncountability strikes yet again! Let’s try to mimick some of the arguments used in the second example in Example 1.40. Note that for any $x \in \R$, the singleton $\{x\}$ is closed (because its complement is open, cf. Definition 1.13) and is therefore an element of $\mathcal{B}(\R)$. If we now take any subset $A \subseteq \R$, then we can trivially write it as a union of all its elements as singletons. It would then seem that it follows from Definition 1.33 iii that also $A \in \mathcal{B}(\R)$. Since this works for any subset of $\R$, it would lead us to conclude that $\mathcal{B}(\R)$ contains all subsets of $\R$ i.e. that it is simply the powerset $\mathcal{P}(\R)$. But this is a contradiction, because we said above that there do exist subsets of $\R$ that are not elements of $\mathcal{B}(\R)$. So there must a mistake in our arguments here. Do you see where it is?

Remark 1.43. In our discussion here, we used the collection of all open subsets of $\R$ to generate $\mathcal{B}(\R)$. But it turns out that this is not the only possible choice. For instance, it is also sufficient to use the collection of all intervals of the form $(-\infty,x]$ for all $x \in \R$ as your generator, cf. Lemma 1.59.

Remark 1.44. Our definition of the Borel $\sigma$-algebra is pretty simple: all it uses is that on $\R$ the concept of open subsets exists. Of course there are many other universes that also have this concept, probably $\R^n$ immediately comes to mind but there are many others as well. On all these universes, the Borel $\sigma$-algebra can be naturally defined as generated by all open subsets.

Exercises

You can now do Exercise 1.9.

1.5.4 A sigma-algebra as a “level of information”

So far in this section we have mainly focused on properties and examples of $\sigma$-algebras defined on any set/universe $\Omega$, and on ways to construct them. At the start of this section we have briefly illustrated our main motivation for introducing them: in advanced probabilistic experiments in particular (cf. Definition 1.31), as we will see we will need them to have a way to define a collection of “valid” events to avoid technical trouble.

While that’s very true, in this final subsection dedicated to $\sigma$-algebras we would like to highlight one particular interpretation of $\sigma$-algebras, one particular way to give them a meaning, namely as representing a certain “level of information”. For this we stick with interpreting the universe $\Omega$ as the outcome space of some experiment (for convenience, as it makes introducing this interpretation easier).

To kick off the discussion, imagine the following: you are a super hero being held hostage in a room. In another room, your evil enemies execute an equally evil probabilistic experiment, with outcome space $\Omega$. The outcome of the experiment, say $\omega \in \Omega$, is observed by your enemies but not by you. You’re keen to find out as much as possible about the outcome of the experiment! Some enemies in the other room are feeling bad for you, and they will pass you some information about that unknown (to you) outcome.

Suppose for the moment that the experiment consists of tossing three coins (maybe not that evil, this one :)), so $\Omega$ consists of all triplets with possible entries H and T. Suppose that two enemies sneak into your room (hopefully they don’t notice each other or things could get ugly!) to give you the following info:

Enemy 1: “Pssst! The first coin is a H!”
Enemy 2: “Pssst! The first coin is a H and the second a T!”

If we now compare the “level of information” (about the unknown outcome) that each of these two statements provides, which one contains more info? You’ll agree that that is the second one no?

How do we formalise this intuition? The trick is to look at the set of all elements in $\Omega$ that are still candidates to be the unknown outcome given the info provided:

Enemy 1: the set of all triplets of the form $(\text{H},\, . \, , \, . \,)$, where the final two entries are either H or T. So that’s 4 triplets/elements of $\Omega$ in total.
Enemy 2: the set of all triplets of the form $(\text{H},\text{T}, \, . \,)$, where the final entry is either H or T. So that’s 2 triplets/elements of $\Omega$ in total.

If we call the first set $A$ and the second $B$, then we see that $B \subset A$: $B$ is contained in $A$ and is strictly smaller than $A$. In this sense, enemy 2 (or: the knowledge that the unknown outcome is an element of $B$) provides strictly more information than enemy 1 (or: the knowledge that the unknown outcome is an element of $A$).

So in general the principle is: if you’re being told that an event $A$ happens i.e. that the unknown outcome is an element of $A$, then the smaller $A$ is the more information that gives you. The two extreme cases are:

if $A=\Omega$ then there is nill information in it for you: that event always happens if an experiment is executed and doesn’t narrow down things at all;
if $A$ is a singleton i.e. $A$ contains only one element, then you’ve just won the lottery of maximal information: after all that event happening means that its one element must exactly be the unknown outcome!

Ok, now here is how $\sigma$-algebras come into this story. Suppose that on your outcome space $\Omega$ (be it the coins above or some other experiment) two $\sigma$-algebras (recall the def from Definition 1.33) are defined: $\mathcal{F}$ and $\mathcal{G}$. And suppose that these two friendly enemies provide you with the following information about the unknown outcome:

Enemy 1 will give you a list of all elements in $\mathcal{F}$ that happen i.e. that contain the unknown outcome (and hence the elements of $\mathcal{F}$ not on that list do not happen).
Enemy 2 does the same thing, but with $\mathcal{G}$ rather than $\mathcal{F}$.

Suppose that $\mathcal{F} \subseteq \mathcal{G}$. Then which enemy gives you the most information? My claim is that it is enemy 2. Here is the argument. For any $A \in \mathcal{F}$, you will learn from enemy 1 whether or not it happens. But since $\mathcal{F} \subseteq \mathcal{G}$, this $A$ is also an element of $\mathcal{G}$ and hence that same bit of information is also present on the list of enemy 2! So enemy 2 gives you at least as much information as enemy 1, and potentially strictly more (depending on the elements of $\mathcal{G}$ that are not in $\mathcal{F}$). So:

Proposition 1.45 (The “level of information” a sigma-algebra represents) Let $\Omega$ be the outcome space of some experiment (and our universe).

We understand/define/interpret the “level of information” that a $\sigma$-algebra $\mathcal{F}$ (cf. Definition 1.33) on $\Omega$ represents as follows: if the experiment is done with unknown outcome $\omega \in \Omega$, and you are being told which events in $\mathcal{F}$ happen, how much information does that give you about that unknown outcome.
Suppose that $\mathcal{F}, \mathcal{G}$ are two $\sigma$-algebras on $\Omega$ satisfying $\mathcal{F} \subseteq \mathcal{G}$ (this is regularly also expressed in words as “$\mathcal{F}$ is a sub-$\sigma$-algebra of $\mathcal{G}$”). Then the larger $\sigma$-algebra (i.e. $\mathcal{G}$) provides at least as much information/at least as high a level of information (about the unknown outcome of the experiment) as the smaller $\sigma$-algebra (i.e. $\mathcal{F}$).

Note that the argument we gave above Proposition 1.45 really depends on one of the two $\sigma$-algebras being contained in the other. If that is not the case, it is impossible to make general statements about which of the two provides more information.

We conclude this subsection with some examples and remarks. We won’t return to this “level of information” shizzle in the rest of this chapter, but it will pop up several times later on in the course!

Remark 1.46. If you’re a soft natured mathematician and find all that super hero vs enemies from above unnecessarily violent ;), then here is another analogy. Imagine that you’re outside during a nice bright night, and that you’re trying to identify where a particular far away star is located. You’ve brought your telescope with you with different lenses. Now think about staring through a lens towards the sky as if you’re looking at $\Omega$ through a $\sigma$-algebra.

A smaller $\sigma$-algebra gives you a coarser view of $\Omega$ like a poor lens gives you a rather blurry view of the sky: you can roughly make out where the star is but you can’t pinpoint it exactly, the best you can do is say “Well the star is somewhere in this pretty large section of the sky but I can’t bring it down to a more precise, smaller section of the sky” (i.e. “Well the outcome of the experiment is somewhere in this pretty large set/event but I can’t bring it down to a more precise, smaller set/event”).

Changing to a larger $\sigma$-algebra (in an inclusive way) gives you a finer view of $\Omega$ like a better lens gives you a sharper view of the sky: you can now make out more precisely where the star is i.e. identify a smaller section of the sky where the star is definitely located, or: you can now see a smaller and hence more precise/more informative set/event that must contain the outcome of the expriment.

Example 1.47 (Some examples of sigma-algebras and the “level of information” that they represent) Let’s look at some of our favourite simple examples from Example 1.34 in this context!

$\mathcal{F}=\{\emptyset,\Omega\}$. This one is often called the trivial $\sigma$-algebra, for good reason! In the above information context, the event $\Omega$ always happens and the event $\emptyset$ never happens. That tells you absolutely nothing about the outcome of the experiment. Useless. This ties in nicely with the fact that it is the smallest possible $\sigma$-algebra on $\Omega$, because Proposition 1.45 tells us, loosely speaking: larger means more info.
The largest possible $\sigma$-algebra on $\Omega$ on the other hand is the powerset $\mathcal{P}(\Omega)$ (cf. Definition 1.27). This one provides perfect/full information (also in line with “larger means more info”!). After all, it also contains all possible singletons i.e. $\{ \omega \}$ for each $\omega \in \Omega$. When the experiment is being done, the list of which events/elements in $\mathcal{P}(\Omega)$ happen will inevitably contain exactly one of these singletons, revealing exactly what the outcome of the experiment was!

Example 1.48 (Finite sigma-algebras and the “level of information” that they represent) I know, I know, these two examples in Example 1.47 still feel a little bit trivial, do we not have anything a little bit more subtle?? Well we do, but the eternal issue is just how annoying it is to fully specify non-trivial $\sigma$-algebras. The one exceptional category is, as we previously saw in Subsection 1.5.2: finite $\sigma$-algebras.

Let $\mathcal{F}$ be a finite $\sigma$-algebra on $\Omega$. Recall from Proposition 1.38 that it must have atoms $A_1, \ldots, A_n$ which also form a (finite) partition of $\Omega$, and that all its elements, besides $\emptyset$ and $A_1, \ldots, A_n$ themselves, take the form of a union of two or more choices from $A_1, \ldots, A_n$. Before doing the “level of information” business in detail, let’s explore a little bit what the nice structure of such a $\sigma$-algebra exactly brings us in this context (that we can’t necessarily do for a general $\sigma$-algebra!).

Suppose for the moment that the experiment is done, and that the outcome is an element of $A_1$. Which elements of $\mathcal{F}$ do then happen? Well $A_1$ of course (tautology), and for instance also $A_1 \cup A_2$, and $A_1 \cup A_4$, and $A_1 \cup A_2 \cup A_5$, and … Indeed, the elements of $\mathcal{F}$ that happen are, besides $A_1$ itself, any possible union of any number of $A_1, \ldots, A_n$ as long as $A_1$ is included in that union as well. All the other elements of $\mathcal{F}$, i.e. $A_2, \ldots, A_n$ and any union of any number of those, do not happen because $A_1, \ldots, A_n$ is a partition and hence they are mutually disjoint: if $A_1$ happens, then none of the $A_2, \ldots, A_n$ happen. Agreed?

Ok, now take a close look at that pile of events that will happen in this scenario. Note that $A_1$ is a subset of each of the events in that pile (as earlier mentioned in Remark 1.39 as well). That means, by the same argument as we used in the example with the enemies at the start of this subsection, that the fact that $A_1$ happens is the most informative statement (the “smallest” of all events that happen), all the other events in $\mathcal{F}$ that also happen don’t add any useful info to the knowledge that $A_1$ happens.

You can repeat this exact argument by going through each of the other atoms $A_2, \ldots, A_n$ one by one and imagining that it contains the outcome.

Now let’s go back to the “level of information” setup from Proposition 1.45: the experiment is being done and you are (only) being told what elements of $\mathcal{F}$ happen. Since the atoms $A_1, \ldots, A_n$ form a partition of $\Omega$, it is guaranteed (by def, cf. Definition 1.23) that exactly one of $A_1, \ldots, A_n$ will happen, say $A_i$. Together with $A_i$, as argued above a whole bunch of other elements of $\mathcal{F}$ will also happen, but these don’t add any further information.

This brings us to the following principle: for a finite $\mathcal{F}$ with atoms $A_1, \ldots, A_n$, the level of information it represents is exactly equivalent to being told which one of the atoms $A_1, \ldots, A_n$ happens (i.e. contains the outcome).

It’s nice to observe that this also means that typically, the information level represented by a finite $\sigma$-algebra drifts somewhere strictly between the “no information at all” of the trivial $\sigma$-algebra (cf. Example 1.47 i) and the “full information” of the powerset (cf. Example 1.47 ii): being told which atom happens is better than being told the trivial statement that $\Omega$ happens, because normally that atom will be a strict subset of $\Omega$, yet not as great as being told which singleton happens because normally that atom will contain multiple elements that are all still candidates for the unknown outcome.

Ok, finally then: suppose that $\mathcal{F}$ and $\mathcal{G}$ are both finite $\sigma$-algebras on some outcome space/universe $\Omega$, with atoms $A_1, \ldots, A_n$ and $B_1, \ldots, B_m$ respectively. If you think about it, you’ll be able to convince yourself that $\mathcal{F} \subseteq \mathcal{G}$ holds if and only if $B_1, \ldots, B_m$ is a finer partition of $\Omega$ than $A_1, \ldots, A_n$ is, i.e. if each $A_i$ either equals one of $B_1, \ldots, B_m$ or otherwise can be written as the union of two or more of the $B_1, \ldots, B_m$.

In the analogy of taking a sheet of paper and cutting it into pieces to obtain a partition: first cut it into $n$ pieces and use these as your partition to generate $\mathcal{F}$. Then further cut at least one of the pieces into two or more pieces, and use that $m>n$ number of pieces as a new partition to generate $\mathcal{G}$. (This creates the strict case $\mathcal{F} \subset \mathcal{G}$, otherwise $\mathcal{F} \subseteq \mathcal{G}$ of course only holds if $\mathcal{F}=\mathcal{G}$ so that they have the same atoms.)

My point is that the conclusion of Proposition 1.45 ii can be quite precisely seen in action here: $\mathcal{G}$ represents at least as high a level of information as $\mathcal{F}$, because (using the above principle) the info it represents is equivalent to being told which one of the $B_1, \ldots, B_m$ happens which is at least as good as being told which one of the $A_1, \ldots, A_n$ happens. Indeed if say $B_i$ happens then there are two possible scenarios:

$B_i$ equals one of the $A_1, \ldots, A_n$ (paper sheet analogy above: $B_i$ is a piece of paper that you did not further cut up in the second round of cutting), in which case it’s an equal level of information,
otherwise if $B_i$ is not one of the $A_1, \ldots, A_n$ then $B_i$ must be a strict subset of exactly one of $A_1, \ldots, A_n$, say $A_j$ (paper sheet analogy: $B_i$ is a piece of paper that appeared in the second round of cutting, where you decided to cut piece $A_j$ from round one further up in smaller pieces) then it’s strictly better information because $B_i \subset A_j$.

Remark 1.49. Some of you may have spotted a pattern in all of the above, maybe in particular in Example 1.48, as follows. If we know that two events $A$ and $B$ happen, then what is the sharpest piece of information we can deduce from that? Well indeed, the smallest set that we are sure will happen is then $A \cap B$, so that best piece of information is: “the outcome of the experiment must be an element of $A \cap B$”.

That principle also naturally translates to Example 1.48: if we know which events in a finite $\sigma$-algebra happen, then the (unique) atom that happens can be found by taking the intersection of all events that happen (you can easily check this from the structure we discussed in that example).

Then the natural question pops up: we have said that a general $\sigma$-algebra, say $\mathcal{F}$, is more unwieldy, ok, fair enough, but can we then not do the same thing there to find a bit more structure: look at all events in $\mathcal{F}$ that happen, and take their intersection to end up with the smallest event that happens as the sharpest piece of info it provides? Well…. no, not in general (with our level of knowledge at this point) at least! The problem is again, sigh, the Drama Queen of Uncountability! This approach is guaranteed to work for finite $\sigma$-algebras, because such an intersection is then necessarily always over at most finitely many events, and then the result is guaranteed to still be an element of that $\sigma$-algebra (cf. Proposition 1.35). However if $\mathcal{F}$ is uncountable, then it may very well be the case that uncountably many events in $\mathcal{F}$ happen, and taking the intersection over all of those is not covered by Proposition 1.35, and therefore we have no guarantee that the end result is actually an event in $\mathcal{F}$ which happens!

Nevertheless, despite that technical difficulty, at an intuitive level/as a way to make sense/think about it, it is perfectly ok to think about the level of info that a $\sigma$-algebra provides about the outcome of the experiment as if you’d take the intersection of all events in that $\sigma$-algebra that happen to come to the smallest set/region of $\Omega$ where the $\sigma$-algebra tells you that the outcome must be in.

1.6 (Probability) measures & measure/probability spaces

On to the final topic of this chapter then, dealing with the second challenge we formulated at the end of Section 1.4! We are working with some measurable space (cf. Definition 1.33) i.e. a pair $(\Omega,\mathcal{F})$ where $\Omega$ is a universe i.e. set and $\mathcal{F}$ a $\sigma$-algebra on $\Omega$.

First looking at the specific interpretation that we use in Probability Theory, where $\Omega$ is the outcome space of some experiment (cf. Definition 1.30) and $\mathcal{F}$ the collection of subsets of $\Omega$ that we consider the (“valid”) events (in this section we’ll finally get round to discussing why this is necessary!), we are looking to define a mathematical object that we can use to represent probabilities of these events.

So what should that look like? Well it should be an object that gives each event i.e. element of $\mathcal{F}$ a value from the interval $[0,1]$ that we interpret as the likelihood that the event happens if we execute the experiment right. Hence, in terms of maths, a mapping⁶ with as domain $\mathcal{F}$ and as co-domain $[0,1]$. Would any such mapping do? Nah, not entirely, there are certain properties you’d like it to have for it to make sense in this context, for instance:

⁶ In this course (at least), the terms “mapping” and “function” are simply used interchangeably

the event $\emptyset$ never happens, and hence you would want it to be mapped to the value/likelihood $0$;
for any events $A, B \in \mathcal{F}$ with $A \subseteq B$, if $A$ happens then $B$ happens as well and hence you’d want that the value/likelihood $B$ gets mapped to is at least as large as the value/likelihood that $A$ gets mapped to right;
for any disjoint events $A, B \in \mathcal{F}$, the event $A \cup B$ happens exactly if either $A$ or $B$ happens, and because they are disjoint we naturally want to have additivity: the value/likelihood $A \cup B$ gets mapped to should equal the sum of the values/likelihoods that $A$ and $B$ get mapped to;
and several others!

Fine! But we can actually step away from the probability interpretation for a bit into the slightly more general measure theory context, in which we see $(\Omega,\mathcal{F})$ as “just” some universe/set with some $\sigma$-algebra defined on it. Also here there are many situations in which you’re interested in, roughly speaking, measuring a particular aspect of each element of your $\mathcal{F}$ (in the widest possible sense) subject to the same desired properties we listed in 1.–4. above. Some (more or less) explicit examples (convince yourself that the above properties indeed hold!):

If $\Omega$ is finite (or countable), the mapping that assigns to each $A \in \mathcal{F}$ the number of elements $A$ contains;
If $\Omega=\R^n$, the mapping that assigns to each $A \in \mathcal{F}$ its $n$-dimensional volume;
If $\Omega$ is any universe and we fix some element $\omega_0 \in \Omega$, the mapping that assigns to each $A \in \mathcal{F}$ the value $1$ if $\omega_0 \in A$ and the value $0$ otherwise;
If $\Omega$ is the set of all continuous functions $\R \to [0,\infty)$, the mapping that assigns to each $A \in \mathcal{F}$ the total number of roots that all functions in $A$ have;
etc. etc. — there’s zillions more!

In general, we call such a mapping $\mu: \mathcal{F} \to [0,\infty]$ a measure ($[0,\infty]$ is defined as $[0,\infty) \cup \{\infty\}$ i.e. all non-negative real numbers plus $\infty$). To use such a guy for assigning probabilities/likelihoods i.e. to make it a probability measure, it should satisfy $\mu(\Omega)=1$, which ensures that the co-domain is $[0,1]$ rather than $[0,\infty]$. It turns out that to ensure that our (probablity) measure has the desirable properties we listed above (1.–4.), the only conditions we need to impose are that $\mu(\emptyset)=0$ and a suitable form of additivity. All (other) desirable properties then very conveniently follow, cf. Theorem 1.51.

Definition 1.50 ((Probability) measures and measure/probability spaces) Consider some measurable space $(\Omega,\mathcal{F})$ (recall from Definition 1.33). A measure on $(\Omega,\mathcal{F})$ is a mapping $\mu: \mathcal{F} \to [0,\infty]$ (where $[0,\infty]:=[0,\infty) \cup \{\infty\}$) satisfying the following conditions:

$\mu(\emptyset)=0$,
countable additivity (or $\sigma$-additivity): if $A_1, A_2, \ldots \in \mathcal{F}$ are mutually disjoint (recall from Definition 1.20) then \[\mu \left( \bigcup_{i=1}^\infty A_i \right) = \sum_{i=1}^\infty \mu(A_i). \tag{1.9}\]

The resulting triplet $(\Omega,\mathcal{F},\mu)$ consisting of a universe, a $\sigma$-algebra and a measure is called a measure space.

Some notes:

Recall that $\emptyset$ and $\Omega$ are guaranteed to be elements of $\mathcal{F}$ (cf. Definition 1.33) and hence also of the domain of $\mu$. This is used to be able to formulate condition i above as well as the next bullet point below.
Note that it follows from Theorem 1.51 i that $\mu(\Omega)$ is the largest value $\mu$ can take. With this mind we have the following two special cases:
- If $\mu(\Omega)<\infty$ then $\mu(A) \in [0,\infty)$ for all $A \in \mathcal{F}$ and we call $\mu$ a finite measure.
- If $\mu(\Omega)=1$ then $\mu(A) \in [0,1]$ for all $A \in \mathcal{F}$ and we call $\mu$ (can you guess it??) a probability measure. As we’re used to from our earlier Prob courses, in this case we often use the symbol “$\P$” rather than “$\mu$” and we typically call the triplet $(\Omega,\mathcal{F},\P)$ a probability space rather than just “measure space”. In this context, the elements of $\mathcal{F}$ are what we consider our (“valid”) events, to which $\P$ assigns a probability/likelihood of happening.
Note that, analogue to the same point in Definition 1.33, if $\mu$ is a measure then Equation 1.9 holds for any finite sequence $A_1, A_2, \ldots, A_n \in \mathcal{F}$ of mutually disjoint events as well. Indeed just extend it to an infinite one by setting $A_{n+1}:=\emptyset$, $A_{n+2}:=\emptyset$, … and apply both conditions above to this infinite sequence to deduce that \[\mu \left( \bigcup_{i=1}^n A_i \right) = \sum_{i=1}^n \mu(A_i).\]

Here are some of the key properties that (probability) measures have — they will be ringing a few bells with you, since you’ll have seen and used most/all of them in your earlier Probability courses! We will leave the proofs up to you in the exercises.

Theorem 1.51 (Properties of (probability) measures) Let $(\Omega,\mathcal{F},\mu)$ be a measure space as defined in Definition 1.50. Then the following holds.

For the first two properties, let $A, B \in \mathcal{F}$.

Suppose that $A \subseteq B$. Then $\mu(A) \leq \mu(B)$. If $\mu(A)<\infty$, then also $\mu(B \setminus A)=\mu(B)-\mu(A)$.
If $\mu(A \cap B)<\infty$ then $\mu(A \cup B)=\mu(A)+\mu(B)-\mu(A \cap B)$.

For the remaining properties, let $A_1, A_2, \ldots \in \mathcal{F}$ (so, a countable sequence).

Subadditivity: it holds that \[\mu \left( \bigcup_{i=1}^\infty A_i \right) \leq \sum_{i=1}^\infty \mu(A_i) \tag{1.10}\] (compare and contrast with Equation 1.9 — note that here $A_1, A_2, \ldots$ are not assumed to be mutually disjoint!)
If $A_1, A_2, \ldots$ is a non-decreasing sequence i.e. $A_1 \subseteq A_2 \subseteq \ldots$ then it has a limit $A$ (in $\mathcal{F}$, cf. Definition 1.33 iii) which we can e.g. express as \[A = \bigcup_{i=1}^\infty A_i\] and we have that \[\mu(A)=\lim_{i \to \infty} \mu(A_i).\]
Similarly, if $A_1, A_2, \ldots$ is a non-increasing sequence i.e. $A_1 \supseteq A_2 \supseteq \ldots$ then it has a limit $A$ (in $\mathcal{F}$, cf. Proposition 1.35 i) which we can e.g. express as \[A = \bigcap_{i=1}^\infty A_i\] and provided that $\mu(A_1)<\infty$ we again have that \[\mu(A)=\lim_{i \to \infty} \mu(A_i).\]

Some notes:

For properties iii–v, recall that we discussed infinite unions and intersections in Definition 1.10.
Note that property iii also holds for a finite sequence $A_1, \ldots, A_n \in \mathcal{F}$: \[\mu \left( \bigcup_{i=1}^n A_i \right) \leq \sum_{i=1}^n \mu(A_i)\] (analogue to what we also did in Definition 1.33 and Proposition 1.35: just extend the sequence to an infinite one by setting $A_{n+1}:=\emptyset$, $A_{n+2}:=\emptyset$, … and apply property iii as well as Definition 1.50 i to it).
Properties iv and v express a notion of “continuity” of a measure: if sets $A_i$ tend to some limit set $A$ (in a monotone fashion), then $\mu(A_i)$ tends to $\mu(A)$.
Properties i, ii and v contain a finiteness condition which is by definition satisfied if $\mu$ is a finite measure or probability measure.

Lovely. The final new concept that we need to introduce in this chapter is the Lebesgue measure, arguably the most prominent less obvious example of a measure as defined in Definition 1.50! We’ll do that in Subsection 1.6.1 below. However before we go there, here are some examples. They make the link between the somewhat more abstract material in this chapter and your Prob knowledge from earlier courses, as well as discussing the still outstanding question of how to construct probability measures in less obvious cases.

Example 1.52 (An example with (infinitely many) probability measures) Ok then, now we have introduced this concept of (probability) measures, let me get one thing out of the way to make a clean connection with the work with probabilities you have done in the past.

First we make the observation that the above Definition 1.50 defines/allows for in principle many possible measures on any measurable space $(\Omega,\mathcal{F})$. In the probability context this means: for any experiment with outcome space $\Omega$ and collection of events $\mathcal{F}$ there are many possible probability measures. But… ehm Kees, shouldn’t we have only one such thing, after all each event has only one unique probability/likelihood right? Well, yes but also no! ;). Let’s discuss.

Consider the simplest possible experiment you could think of: toss a coin, with outcome space $\Omega=\{\text{H},\text{T}\}$. Let the $\sigma$-algebra $\mathcal{F}$ simply be the powerset (that’s fine here, since this is a simple experiment, recall from Definition 1.31): \[\mathcal{F}=\big\{ \emptyset, \{\text{H}\}, \{\text{T}\}, \Omega \big\}.\]

Now, let’s first strictly look at Definition 1.50 to wonder what the possible probability measures are on this space (we’ll use “$\P$” rather than “$\mu$”). As you can readily check, for any $\alpha \in [0,1]$ the mapping $\P_\alpha: \mathcal{F} \to [0,1]$ defined as \[\P_\alpha(\emptyset)=0, \quad \P_\alpha(\{\text{H}\})=\alpha, \quad \P_\alpha(\{\text{T}\})=1-\alpha, \quad \P_\alpha(\Omega)=1 \tag{1.11}\] is a probability measure on $(\Omega,\mathcal{F})$ as defined in Definition 1.50. And vice versa, each possible probability measure on $(\Omega,\mathcal{F})$ is one of those guys (you may well understand why immediately, if not then we’ll see an argument in Example 1.54 below). Note that even though this is the simplest/smallest outcome space you could hope for, there already exist infinitely many (in fact uncountably many, since the “index set” $[0,1]$ is uncountable, cf. Example 1.17) probability measures on this space! On larger spaces this will only get worse, so we should quickly set aside any expectation that we’ll ever find a situation with only one, or even only finitely many possible probability measures!

And this is perfectly ok — in fact a fundamental concept in Financial Maths makes use of this feature and in particular the fact that you can “switch” from one probability measure to another (cf. Section 4.6).

But, how does this tie in with your previous Prob experience, in which each event has a single probability attached to it? The answer is as follows. Each possible probability measure describes a possible “reality” for your experiment. For instance in the above example, if we choose $\alpha=1/4$ then we are dealing with a coin that yields H with prob $1/4$ and T with prob $3/4$ (just plug $\alpha=1/4$ into Equation 1.11). Similarly if we choose $\alpha=1/2$ then we are dealing with a fair coin, and for $\alpha=0$ then it’s a coin with two tails!

So this whole spectrum of probability measures available allows you to dial up or down the probability of any of the events as you see fit (though $\emptyset$ always has prob $0$ and $\Omega$ always has prob $1$, per Definition 1.50). In a typical application in your previous Prob courses, you were given a bit of extra info that specifies the “reality” i.e. which probability measure is the right one to use. For instance “Consider the experiment of tossing a fair coin, …”, which tells you that $\P_{1/2}$ is the right one to use. Doesn’t mean that the others suddenly no longer exist of course, but rather just that $\P_{1/2}$ is the one that describes the correct “reality” that we want to work with and is hence the one to do your computations with.

Example 1.53 (Probability measures vs empirical probabilities) This one is just to very quickly recall a connection that you’ll already be aware of at some level, nevertheless good to reiterate! If you have a certain concrete probabilistic experiment in mind, then via Definition 1.30 and Definition 1.50 you can construct a probability space $(\Omega,\mathcal{F},\P)$ that is a mathematical representation of that experiment. As discussed in Example 1.52, in particular you want to choose the “right” probability measure on $(\Omega,\mathcal{F})$ out of the many possible ones, the one that reflects the “reality” of the experiment as we called it there.

But what does that exactly mean, and how does it tie in with the real world side of having a concrete experiment that gets executed? Well intuitively we know that if we toss a fair coin a lot of times, then in approximately 50% of cases the outcome will be H and in approximately 50% of cases it will be T. That “experience fact” leads us to believe that $\P_{1/2}$ (in the notation of Example 1.52) is the “right” probability measure to use in this situation. More precisely, if we toss a coin $n$ times and observe $m \in \{0,1,\ldots,n\}$ times H, and that is all we know, then our best guess is that the coin yields H with probability $m/n$ and T with probability $(n-m)/n$. These are called empirical probabilities (or the empirical probability measure, as this translates to the prob measure $\P_{m/n}$ from Equation 1.11).

However these empirical probabilities are of course always subject to a degree of error/uncertainty — I mean, a fair coin with “right” probability measure $\P_{1/2}$ could produce H $n$ times out of $n$ tosses, sure this is very unlikely if $n$ is large but never entirely impossible! The only relationship between the “right” probability measure and empirical probabilities that is not subject to any error/uncertainty is provided by the always amazing Law of Large Numbers, which formulated generally tell us: if $A \in \mathcal{F}$ is an event and $\P$ the “right” probability measure, we execute the experiment $n$ times, and denote by $m(n) \in \{0,1,\ldots,n\}$ the number of times that the event $A$ is observed to happen, then we have that \[\P(A)=\lim_{n \to \infty} \frac{m(n)}{n}.\] That is: we’d need to execute the experiment a practically impossible infinite number of times to deduce the exact value $\P(A)$ from our observations, and so in practice the best we can do is to collect as many observations as we can and use these to compute an approximation of $\P(A)$!

Example 1.54 (Probability measures on simple vs advanced experiments) Recall that in Definition 1.31 we set up the distinction between simple and advanced probabilistic experiments, and we warned (several times) that the latter category is technically a lot more challenging in certain aspects. We have come to a point where we can finally start to illustrate this a little bit! We do this in the context of a probability space $(\Omega,\mathcal{F},\P)$ for our experiment, but that’s only for convenience, the conclusions are equally valid in the slightly more general measure theory context i.e. for a general measure space $(\Omega,\mathcal{F},\mu)$ (cf. Definition 1.50).

The simple experiment case. Suppose that we’re dealing with a simple experiment, i.e. a countable outcome space $\Omega$. Recall (cf. Definition 1.16) that this means that we can list its elements/the possible outcomes the of the experiment: $\Omega=\{\omega_1, \omega_2, \ldots\}$ (we focus on the infinite case, the finite case is analogous). Let’s take for our $\sigma$-algebra $\mathcal{F}$, i.e. our collection of (“valid”) events, simply the powerset (cf. Definition 1.27).

In this case, everything is nice and smooth. It is even quite straightforward to characterise all possible probability measures on $(\Omega,\mathcal{F})$ i.e. all possible mappings $\P: \mathcal{F} \to [0,1]$ satisfying the conditions of Definition 1.50. This relies on the following observation only. Let $A \in \mathcal{F}$ i.e. any subset of $\Omega$. Then we can trivially express $A$ as the union of all its elements in singleton form. For example: $\{\omega_1,\omega_5\}=\{\omega_1\} \cup \{\omega_5\}$, $\{\omega_5,\omega_{10},\omega_{15}, \ldots \}=\{\omega_5\} \cup \{\omega_{10}\} \cup \{\omega_{15}\} \cup \ldots$ etc., you catch my drift. It’s clumsy to write down, but maybe \[A = \bigcup_{i: \ \omega_i \in A} \{ \omega_i \}.\] And crucially, because there are only countably many elements in $\Omega$, it’s guaranteed that this is always a union of only countably many singletons. Further these singletons are clearly mutually disjoint. Hence it follows from countable additivity (cf. Definition 1.50) that any probability measure $\P: \mathcal{F} \to [0,1]$ on $(\Omega,\mathcal{F})$ satisfies \[\P(A)=\P \left( \bigcup_{i: \ \omega_i \in A} \{ \omega_i \} \right) = \sum_{i: \ \omega_i \in A} \P(\{ \omega_i \}) \tag{1.12}\] or in words: the probability of the (any!) event $A$ equals the sum of the probabilities of the outcomes $A$ contains.

This means that if we know/choose the probability of each outcome in $\Omega$, then the whole probability measure is immediately fully and uniquely specified. And vice versa: any possible probability measure is tied to the probabilities it assigns to each outcome in $\Omega$. So there is a one-to-one correspondence between all possible probability measures on $(\Omega,\mathcal{F})$ on the one hand and all possible sequences $a_1, a_2, \ldots$ of non-negative numbers that sum up to $1$ on the other hand (of course, in such a sequence $a_i$ represents the probability of the outcome $\omega_i$, for $i=1,2,\ldots$). This gives us a full and relatively simple representation of all possible probability measures on this space!

Note that Example 1.52 is of course a specific example of an experiment in this category, after all there $\Omega=\{\text{H},\text{T}\}$ and so the previous paragraph tells us that all possible probability measures in that example are characterised by all possible pairs of non-negative numbers $a_1, a_2$ (the prob of outcome H resp. outcome T) that sum up to $1$. Or, equivalently, numbers $\alpha$ and $1-\alpha$ for $\alpha \in [0,1]$ as we formulated it in Example 1.52!

And as final note: this discussion also shows why the most elementary Prob Theory course, where only simple experiments are considered, can be done without any proper discussion of probability measures at all: all you really need is the principle that probabilities of individual outcomes generate probabilities of events via the addition rule Equation 1.12.

The advanced experiment case. Now let’s look at an example with an uncountable outcome space $\Omega$. Consider the experiment of rolling dice. Clearly if we roll $n$ of them, then $\Omega$ is finite ($6^n$ possible outcomes after all) and we are in the comforting simple experiment setting from above. Let’s push this experiment, seemingly just a little bit, further: suppose that we roll infinitely many fair dice. Then each possible outcome i.e. element of $\Omega$ is an infinite sequence, all entries of which are one of ⚀, ⚁, ⚂, ⚃, ⚄, ⚅ (cute fancy symbols eh! ;)). This pushes us over the edge into the wild world of advanced experiments, as the outcome space $\Omega$ consisting of all possible such sequences is indeed uncountable (cf. Example 1.17 vii).

The two questions we’d have to answer are, as in the simple case above: what about a $\sigma$-algebra $\mathcal{F}$ consisting of subsets of $\Omega$ making up our (“valid”) events, and what about probability measures on this space? Well… it’s complicated. ;).

Firstly, we cannot simply take the powerset as our $\sigma$-algebra, because if we do then we’ll get into trouble trying to construct probability measures (this is a bit easier to explain the context of the Lebesgue measure, so we will do that in Subsection 1.6.1 below).

Secondly, the naive/straightforward way of constructing probability measures as in the simple case breaks down here. Indeed consider for instance the probability measure $\P$ that represents the “reality” in which the dice are fair, assuming for the moment that it indeed exists. Then my claim is that for every possible outcome $\omega \in \Omega$ (so $\omega$ is any one of these infinite sequences with entries from ⚀, ⚁, ⚂, ⚃, ⚄, ⚅) we have that \[\P(\{\omega\})=0. \tag{1.13}\] Why? Well, fix $\omega \in \Omega$. For any $n=1,2,\ldots$, let $A_n$ be the event that contains all the outcomes in $\Omega$ whose first $n$ entries are the same as those in $\omega$, and all later entries can be anything. Then the probability that $A_n$ happens is clearly $(1/6)^n$ (the probability of rolling a given sequence with $n$ dice, the other rolls don’t matter). And also clearly $\{\omega\} \subseteq A_n$, so that Theorem 1.51 i yields $\P(\{\omega\}) \leq (1/6)^n$. Since this inequality holds for all $n=1,2,\ldots$, we indeed also get that \[\P(\{\omega\}) \leq \lim_{n \to \infty} \left( \frac{1}{6} \right)^n=0.\] So: each possible outcome of our experiment has probability $0$! Chew chew chew….

But ehm Kees, you know, shouldn’t the probabilities of all the possible outcomes add up to $1$?? Well, that principle fundamentally relies on two assumptions: that $\P(\Omega)=1$ (which is still true, cf. Definition 1.50) and that $\Omega$ can be expressed as a countable union of all possible outcomes as singletons, so that we can apply countable additivity (cf. Definition 1.50 ii). This is, indeed, exactly the line of reasoning we used in the simple case above and it is perfectly valid there. But, you guessed it, this reasoning simply doesn’t apply here because $\Omega$ is uncountable, and so there is no contradiction here! (No, don’t get scared, that weird sound you’re hearing is just the Witch of Uncountability arriving — I guess that her broom needs a bit of grease.)

There are two important consequences of Equation 1.13:

There exist events that are not the empty set (so it is not impossible for them to happen) that nevertheless have probability $0$. Such events are called null events. As Equation 1.13 shows, all singletons are null events. But also any countable event $A$ (such as any event containing only finitely many outcomes) is a null event: just apply countable additivity as in Equation 1.12 together with Equation 1.13. It follows that the only events with strictly positive probability are uncountable!

There is also an “other side” to this observation: there exist events strictly smaller than $\Omega$ (so they are not guaranteed to happen) that nevertheless have probability $1$. For instance, let $N$ be any null event and set $A:=\Omega \setminus N$. Then we get from Theorem 1.51 i and Definition 1.50: \[\P(A)=\P(\Omega \setminus N)=\P(\Omega)-\P(N)=1-0=1.\] Such events we call almost sure events.

These concepts pop up quite a lot going forward, and so for clarity we put them together in the below Definition 1.55.
While in the simple case knowing the probabilities of the individual outcomes was sufficient to derive the probability of any event (and the challenge of constructing a probability measure simplifies to giving the individual outcomes probabilities), that is clearly not the case here! Knowing Equation 1.13 is enough to deduce the probability of every countable event $A \in \mathcal{F}$ (namely, $\P(A)=0$, as we also used in 1.) but it tells us nothing about what $\P(A)$ is for uncountable events $A \in \mathcal{F}$! And, yeah, what should it tell us, that these all have probability $0$ as well? Now that would be a real contradiction, because we do know that $\P(\Omega)=1$ (cf. Definition 1.50)! So, the challenge of constructing a probability measure cannot be simplified away here by looking at individual outcomes!

The advanced experiment case: constructing $\sigma$-algebras and probability measures [⛔]. Ok, blahblah Kees, but how do we then actually go about properly constructing (and hence proving existence!) a $\sigma$-algebra and a probability measure in the case of such an advanced experiment?? There are (at least) two options here that I do want to briefly highlight to you, to at least give you the confidence that such objects do exist as we go forward to use and work with them, but on the other hand they are a bit technical and out of scope for this course (we have to draw the line somewhere sadly!) and you shouldn’t worry about any of the remainder below from an exam perspective. We illustrate the ideas via the same experiment we talked about above: rolling infinitely many dice.

It is possible, for this and other types of experiments that consist of repeating the same (random) action with equally likely outcomes infinitely often, to “embed” it into another probability space, namely $([0,1],\mathcal{B}([0,1]),\lambda)$ where $\mathcal{B}([0,1])$ is the Borel $\sigma$-algebra on the universe $[0,1] \subseteq \R$ (cf. Subsection 1.5.3) and $\lambda$ the Lebesgue measure which we’ll discuss in Subsection 1.6.1 below. Indeed, take any outcome $\omega \in \Omega$ i.e. an infinite sequences with entries from ⚀, ⚁, ⚂, ⚃, ⚄, ⚅, and write it as an infinite sequence $\omega=(\omega_1,\omega_2,\ldots)$ with entries from $0,\ldots,5$ rather (so ⚀ becomes $0$, ⚁ becomes $1$ etc.). Then $\omega$ can be associated with a real number $x_\omega$ in the universe $[0,1]$ given by \[x_\omega := \sum_{i=1}^\infty \frac{\omega_i}{6^i} \in [0,1]. \tag{1.14}\] Note that the sequence $\omega=(0,0,\ldots)$ (so rolling only ⚀’s) gets mapped to the real number $x_\omega=0$, while the sequence $\omega=(5,5,\ldots)$ (so rolling only ⚅’s) gets mapped to the real number $x_\omega=1$ (Geometric series eh!).

Using this association, with any Borel set $B \in \mathcal{B}([0,1])$ we can associate a subset $A$ of $\Omega$ by checking for which $\omega \in \Omega$ the associated $x_\omega$ is an element of $B$: \[A:=\{ \omega \in \Omega \, | \, x_\omega \in B \}\] and collecting all these subsets gives a $\sigma$-algebra $\mathcal{F}$ on $\Omega$ to work with as well as a probability measure $\P$ on $(\Omega,\mathcal{F})$ by setting $\P(A):=\lambda(B)$. Obviously omitting some technical details here!

Note how beautifully this construction works! For instance, for any $\omega \in \Omega$ fixed, we get that \[\P(\{\omega\})=\lambda(\{x_\omega\})=0,\] consistent with Equation 1.13 (where we used that the Lebesgue measure of a singleton equals $0$, cf. Subsection 1.6.1). And take for instance the probability of the event $A$ that the first die yields ⚀. Observe that if we translate all $\omega \in A$ to their associated real number Equation 1.14, then the largest real number we’ll find is generated by the outcome $(⚀,⚅,⚅,\ldots)$ with value \[\sum_{i=2}^\infty \frac{5}{6^i}=\frac{1}{6},\] which shows that \[A=\{ \omega \in \Omega \, | \, x_\omega \in [0,1/6] \}\] and hence \[\P(A)=\lambda([0,1/6])=1/6-0=1/6\] (where we again used a property of the Lebesgue measure that we will discuss in Subsection 1.6.1), indeed the value you’d hope to find!

As a final point, maybe you had noted this already, but at an intuitive level also the nature of the Lebesgue measure is exactly what we need here to make sure that the $\P$ we construct represents the “reality” of rolling fair dice: the possible outcomes of rolling $n$ fair dice are all equally likely, this should remain preserved in our $\P$ if we roll infinitely many of them (in a limiting sense only, after all the actual probs are all $0$ as we saw in Equation 1.13!). And since the associated real numbers are nicely uniformly/evenly distributed throughout $[0,1]$, you want a measure on $\mathcal{B}([0,1])$ that represents the “reality” of picking a number from that interval at random with all numbers “equally likely”, exactly what the Lebesgue measure does!
An alternative, and more flexible/general approach, is to make more agressively use of the observation that our experiment consists a repeating a (random) action (rolling a die) infinitely often. If we write $\Omega_0$ for the outcome space of rolling a single die, i.e. $\Omega_0=\{⚀, ⚁, ⚂, ⚃, ⚄, ⚅\}$, then the outcome space of our actual experiment can be written as an infinite Cartesian product (cf. Definition 1.12): $\Omega=\Omega_0^\infty$.

As an intermediate step, consider the experiment of rolling two dice only, with outcome space $\Omega=\Omega_0 \times \Omega_0$. This is of course a “simple” experiment as we already dealt with earlier, but we include it here to illustrate the ideas. Assuming that you can define/construct a $\sigma$-algebra $\mathcal{F}_0$ on $\Omega_0$ and probability measure $\P_0$ on $(\Omega_0,\mathcal{F}_0)$ (which is easy here, since rolling a single die is a “simple” experiment), there is a standard procedure that deals with $\Omega=\Omega_0 \times \Omega_0$, as follows. First we construct what we call the product $\sigma$-algebra $\mathcal{F}$ on $\Omega$, which is the $\sigma$-algebra generated by (cf. Definition 1.36) all subsets of $\Omega$ that take the form of a Cartesian product of a set from $\mathcal{F}_0$ (concerning the first die) with another set from $\mathcal{F}_0$ (concerning the second die): \[\mathcal{F}=\sigma \big( \{ A_1 \times A_2 \subseteq \Omega \, | \, A_1 \in \mathcal{F}_0, \ A_2 \in \mathcal{F}_0 \} \big).\] Next we will construct the so-called product probability measure $\P$ on $(\Omega,\mathcal{F})$ which satisfies \[\P(A_1 \times A_2)=\P_0(A_1) \P_0(A_2) \quad \text{for all } A_1 \in \mathcal{F}_0, \ A_2 \in \mathcal{F}_0. \tag{1.15}\] Note that Equation 1.15 is a property you definitely want, as it reflects that the two dice act independently of each other (the correct “reality”). For instance if we take $A_1=\{⚀, ⚂, ⚄\}$ and $A_2=\{⚄, ⚅\}$ and plus these into Equation 1.15, then Equation 1.15 reads as follows: the probability that the first die yields an odd result and the second a result that is at least $5$ equals the probability that the first die is yields an odd result multiplied by the probability that the second die is at least $5$.

However, Equation 1.15 does not fully define the product measure, as not every element in $\mathcal{F}$ will be of the product form $A_1 \times A_2$ (recall from the discussion in Subsection 1.5.1, that normally a generated $\sigma$-algebra is a rather implicit beast which contains “padding”: extra subsets of unknown form/nature to ensure we actually end up with a $\sigma$-algebra). An example of such a subset is the event that the first die yields a larger result than the second die. Rather, a deep and fundamental result from measure theory called Carathéodory’s extension theorem is needed to establish that a (unique) probability measure on $(\Omega,\mathcal{F})$ with the property Equation 1.15 indeed exists. Ok, now our probability space $(\Omega,\mathcal{F},\P)$, for the case of rolling two dice, is fully constructed using the product space technique!

As a next step, if we can do it for two dice, then we can also do it for any finite number $n=3,4,\ldots$ of dice by extending the above in the obvious way (i.e. using the associative nature).

Now we can finally step up to the case we were actually interested in, with the infinitely many dice rolls! Firstly, to construct a $\sigma$-algebra $\mathcal{F}$ on the outcome space $\Omega$, inspired by the intermediate step we took, you may be tempted to think that we should consider infinite Cartesian products i.e. subsets of $\Omega$ of the form $A_1 \times A_2 \times \ldots$, where each $A_i$ specifies outcomes for the $i$-th die. But it turns out that that doesn’t work. Instead, we need to consider so-called cylinder sets in which you specify outcomes for any choice of the dice as long as you involve only finitely many of them. That is to say, if we write $\omega \in \Omega$ as $\omega=(\omega_1,\omega_2,\ldots)$ then a cylinder set $C$ is of the form \[C=\{ \omega \in \Omega \, | \, \omega_{i_1} \in A_1, \ \omega_{i_2} \in A_2, \ldots, \omega_{i_n} \in A_n \} \tag{1.16}\] for any choices $n=1,2,\ldots$, $i_1, i_2, \ldots, i_n \in \{1,2,\ldots\}$ and $A_1, \ldots, A_n \in \mathcal{F}_0$. In words: an outcome $\omega \in \Omega$ is an element of $C$ if the $i_1$-st die roll is an element of $A_1$, the $i_2$-st die roll is an element of $A_2$, …, the $i_n$-st die roll is an element of $A_n$. Note that Equation 1.16 does have the form of an infinite Cartesian product: it equals $B_1 \times B_2 \times \ldots$, where $B_{i_j}=A_j$ and $B_k=\Omega_0$ for all $k \in \{1,2,\ldots\} \setminus \{ i_1, i_2, \ldots, i_n \}$.

For the $\sigma$-algebra $\mathcal{F}$ on $\Omega$, we take the $\sigma$-algebra generated (cf. Definition 1.36) by these cylinder sets — which then also contains subsets/events like “the second die yields a higher result than the tenth die” and “at least one of the die will yield ⚅” which are not cylinder sets. If you’re not convinced about the latter, then observe that for any cylinder set there is an $m=1,2,\ldots$ so that $\omega \in C$ can only depend on the first $m$ dice rolls recorded in $\omega$ (in Equation 1.16 $m=i_n$), but it is of course not always possible to express in terms of the first $m$ dice rolls whether $\omega$ is an element of “at least one of the die will yield ⚅”: if none of the first $m$ dice rolls yield ⚅ then this depends on the dice rolls after the first $m$!

Further, we can again appeal to the majestic Carathéodory’s extension theorem to establish the existence of a (unique) probability measure $\P$ on $(\Omega,\mathcal{F})$ which represents the right “reality” in the sense that for any cylinder set as in Equation 1.16 it holds that \[\P(C)=\P_0(A_1) \P_0(A_2) \cdots \P_0(A_n).\]

There we are: a fully constructed probability space $(\Omega,\mathcal{F},\P)$! :).

As a final note, a key reason why this approach is more general than the “embedding” technique from approach 1. above is that it allows for chaining together any (random) actions as your experiment. For instance your experiment could consist of rolling a die, tossing a coin, and choosing a real number from $[0,1]$, so that your possible outcomes are triplets and your outcome space $\Omega$ is the Cartesian product of three distinct spaces. And it even allows for chaining uncountably many (random) actions i.e. dealing with outcome spaces that are a Cartesian product over uncountably many spaces! Later on, as we start to talk about stochastic processes in continuous time, this will provide the crucial scaffolding we need at the technical end to make things work!

If you’re interested in more details than I’ve been able to squeeze into the above, here are some suggestions for further reading: Chapter IV in Stroock (1994), Chapter 8 in Williams (1991) and Chapter 1 in Kallenberg (1997).

Definition 1.55 (Null events and almost sure events) Let $(\Omega,\mathcal{F},\P)$ be a probability space (cf. Definition 1.50). An event $A \in \mathcal{F}$ is called a null event if $\P(A)=0$ and an almost sure event if $\P(A)=1$. We also sometimes say “$\P$-null event” and “$\P$-almost sure event” if we want to stress the role of $\P$.

In the case of a simple experiment i.e. when $\Omega$ is countable (cf. Definition 1.30), typically all the possible outcomes have strictly positive probability (i.e. $\P(\{\omega\})>0$ for all $\omega \in \Omega$) in which case $\emptyset$ is the only null event and $\Omega$ the only almost sure event. However in the case of an advanced experiment i.e. when $\Omega$ is uncountable, this is no longer true, as we saw for example in Example 1.54 above!

Little aside: obviously, $\Omega$ is the only event that is always guaranteed to happen, and though it fits our definition of “almost sure event”, a better term for that specific one would be “sure event”. Just read “almost sure event” in general as “at least almost sure events”!

Exercises

You can now do Exercise 1.10–Exercise 1.12.

1.6.1 The Lebesgue measure

As we mentioned earlier on in this chapter, one particular example of a universe $\Omega$ that is close to our heart is of course the good old real line $\R$. We have in Subsection 1.5.3 talked about possible $\sigma$-algebras on the universe $\R$, and came up with the Borel $\sigma$-algebra $\mathcal{B}(\R)$ as a particularly prominent one.

Now, in this section we have introduced the concept of measures in Definition 1.50. Surely we can think of a nice measure to use on our $\R$ as well! What would be natural thing to “measure” about subsets of $\R$? Well, if we start by looking at intervals, say the closed ones $[a,b]$ only for the moment, their length seems like a nice choice — if we look at the requirements in Definition 1.50, then understanding the empty set to have length $0$ would not shock your aunt at Christmas dinner too much, and it seems to play nicely with that countable additivity requirement, at least for finitely many disjoint intervals i.e. $[a_1,b_1]$, …, $[a_n,b_n]$ a natural understanding/definition of the length of their union is the sum of the interval lengths i.e. $b_1-a_1+\ldots+b_n-a_n$ no?

Ok! Let’s see if we can make some true chocolate out of this: Definition 1.50 requires us to come up with a $\sigma$-algebra $\mathcal{F}$ on the universe $\Omega=\R$ (containing at least all closed intervals $[a,b]$) and a mapping, say $\lambda: \mathcal{F} \to [0,\infty]$ that we intend to give the “length” of the elements of $\mathcal{F}$, and which for intervals $[a,b]$ we can explicitly specify as: \[\lambda([a,b])=b-a. \tag{1.17}\] Note that another very natural and hence non-negiotable property of “length” is that it is translation invariant, i.e. if you pick up a subset of $\R$ and move it up or down the real line then it should keep the same length. Indeed that clearly holds true for Equation 1.17: for any $c \in \R$ \[\lambda([c+a,c+b])=b-a=\lambda([a,b]).\]

Note that that the collection of all closed intervals is not a $\sigma$-algebra: it clearly doesn’t satisfy conditions i and ii of Definition 1.33. So we’ll need to be bold and try to make this measure work on a larger collection of subsets that is a $\sigma$-algebra as that is what Definition 1.50 requires. But, you know, can’t we just do away with all this $\sigma$-algebra nonsense, can we not just extend $\lambda$ in a sensible way to all subsets of $\R$ i.e. work on the powerset $\mathcal{P}(\R)$ (cf. Definition 1.27)? I mean, probably any subset of $\R$ can at least be approximated by unions/intersections of closed intervals, and keeping in mind the “continuity” properties that we want a measure to have anyway (recall Theorem 1.51 iv and v) that should allow us to extend Equation 1.17 in a consistent way to any subset of $\R$, thereby also preserving the translation invariance no?

That’s very tempting to think indeed! But along comes, about a century ago, an Italian chap called Giuseppe Vitali with his infamous Vitali sets. A Vitali set $V$ is an explicitly constructed (albeit making use of the Axiom of Choice) subset of $\R$ with the following property. There exist countably many, disjoint subsets $V_1, V_2, \ldots \subseteq \R$, each of which is a translation of the original $V$, so that \[[0,1] \subseteq \bigcup_{i=1}^\infty V_i \subseteq [-1,2]. \tag{1.18}\] Now, suppose that we are able to define a measure $\lambda: \mathcal{P}(\R) \to [0,\infty]$ that is translation invariant and satisfies Equation 1.17. By countable additivity (cf. Definition 1.50 ii) and translation invariance we then have that \[\lambda \left( \bigcup_{i=1}^\infty V_i \right)=\sum_{i=1}^\infty \lambda(V_i)=\sum_{i=1}^\infty \lambda(V).\] Applying this together with Theorem 1.51 i and Equation 1.17 to Equation 1.18 we get to \[1 \leq \sum_{i=1}^\infty \lambda(V) \leq 3.\] That doesn’t look very healthy :(. Indeed, if $\lambda(V)=0$ then the summation equals $0$ and hence we violate the first inequality, while if $\lambda(V)>0$ (obviously the only other option) then the summation equals $\infty$ and hence we violate the second inequality… Conclusion? The only possible one that we are forced to draw: there does not exist a measure $\lambda: \mathcal{P}(\R) \to [0,\infty]$ that is translation invariant and satisfies Equation 1.17!

Where does this leave our noble endeavour? Well we wouldn’t want to compromise on Equation 1.17 or on translation invariance, these are the minimal requirements for a sensible notion of “length”. The only other ingredient on the table is the $\sigma$-algebra we’re using. Maybe it’s possible to take a strictly smaller $\sigma$-algebra $\mathcal{F}$ than the powerset, one that only contains “sensible enough” subsets of $\R$ where we don’t run into problems like the above? Let’s try. What would be our most safe bet? Well $\mathcal{F}$ should contain at least all the closed intervals $[a,b]$ — it would be very weird if our concept of “length” wouldn’t be able to deal with those. The smallest step up from there to a $\sigma$-algebra would be the $\sigma$-algebra generated by (cf. Definition 1.36) these closed intervals then, i.e. \[\mathcal{F} = \sigma \big( \{ [a,b] \, | \, a \leq b \} \big).\]

It turns out that that $\sigma$-algebra $\mathcal{F}$ is nothing but the Borel $\sigma$-algebra $\mathcal{B}(\R)$! Here’s the argument for that:

$\mathcal{F} \subseteq \mathcal{B}(\R)$. Any interval $[a,b]$ is a Borel set (cf. Definition 1.41) and since $\mathcal{F}$ is by def (cf. Definition 1.36) the smallest $\sigma$-algebra containing all these intervals, it follows that $\mathcal{F} \subseteq \mathcal{B}(\R)$.
$\mathcal{B}(\R) \subseteq \mathcal{F}$. Consider the collection of intervals \[\mathcal{C}=\big\{ (-\infty,x] \, | \, x \in \R \big\}.\] Fix any $x \in \R$ and observe that \[(-\infty,x]=\bigcup_{i=1}^\infty [x-i,x]\] (if you’re not convinced, you can use the same arguments as we used in Example 1.11 for instance). Note that on the right hand side we have a (countable) union of closed intervals, and Definition 1.33 iii guarantees that this union is an element of $\mathcal{F}$. Hence also $(-\infty,x] \in \mathcal{F}$ i.e. $\mathcal{C} \subseteq \mathcal{F}$. But $\mathcal{C}$ is a generator of $\mathcal{B}(\R)$ (cf. Lemma 1.59), and hence by def (cf. Definition 1.36) $\mathcal{B}(\R)$ is the smallest $\sigma$-algebra containing all elements of $\mathcal{C}$. Since $\mathcal{F}$ is a $\sigma$-algebra that also contains all elements of $\mathcal{C}$, it follows that $\mathcal{B}(\R) \subseteq \mathcal{F}$.

Ok, so this then culminates in the following question: does there exist a measure (cf. Definition 1.50) $\lambda: \mathcal{B}(\R) \to [0,\infty]$ that is translation invariant and satisfies Equation 1.17? Or will we encounter any Borel sets that blow it all up, like the Vitali sets did above? Reassuringly the answer is yes it does exist! Obviously that needs a proof. We won’t go into such details here, but you can actually explicitly construct it, see e.g. Chapter II in Stroock (1994) if you’re interested in more.

This measure is called the Lebesgue measure and is one of the most prominent and influential measures out there, as we’ll see glimpses of as we go forward. Note that even though it “only” covers Borel sets, that collection is quite rich (recall from Definition 1.41) and is perfectly fine for every day use!

As a little aside: the above also illustrates what we’ve been going on about earlier in this chapter, and what was our motivation to introduce $\sigma$-algebras in the first place: in general, in the case of an advanced experiment i.e. uncountable outcome space/universe (cf. Definition 1.31), it is simply impossible to construct (half decently behaved) measures on the collection of all subsets, and we are simply forced to work on smaller collections of subsets only!

One more thing before we write down the proposition covering the key points. Regulary we are interested in an interval $I \subseteq \R$ as our universe rather than the whole of $\R$. Recall from Definition 1.41 that on such an interval we have also defined a Borel $\sigma$-algebra, denoted by $\mathcal{B}(I)$. Since $\mathcal{B}(I) \subseteq \mathcal{B}(\R)$, our Lebesgue measure on $(\R,\mathcal{B}(\R))$ automatically “induces” a Lebesgue measure on $(I,\mathcal{B}(I))$⁷ which (of course!) then also satisfies Equation 1.17 for intervals $[a,b] \subseteq I$ and is translation invariant⁸.

⁷ It’s as trivial as you think it is: just like the function (say) $f:\N \to \N$ given by $f(x)=x$ “induces” a function on any subset of that domain $\N$, for instance $f:\{0,1,2\} \to \N$ given by $f(x)=x$

⁸ For the sharp thinkers among you: this is one way to define the Lebesgue measure on $(I,\mathcal{B}(I))$. Another way is to start from scratch with $(I,\mathcal{B}(I))$ and just construct the Lebesgue measure on this space independently i.e. following the same “from scratch” method as you would do on $(\R,\mathcal{B}(\R))$. You would hope that both methods result in the same measure, otherwise we have some thinking to do about which one is then the “right” one etc. The good news is that indeed both methods yield the same end result

Proposition 1.56 (The Lebesgue measure and some of its properties) On the measurable space $(\R,\mathcal{B}(\R))$ (recall from Definition 1.33 and Definition 1.41), a measure (cf. Definition 1.50) $\lambda: \mathcal{B}(\R) \to [0,\infty]$ can be defined that gives (a generalised notion of) the length of Borel sets in the sense that it is translation invariant and satisfies for any interval $[a,b]$ with $a \leq b$ (which is indeed a Borel set, recall from Definition 1.41) \[\lambda([a,b])=b-a. \tag{1.19}\] We call $\lambda$ the Lebesgue measure.

For any interval $I \subseteq \R$, on the measurable space $(I,\mathcal{B}(I))$ we automatically also get a Lebesgue measure $\lambda: \mathcal{B}(I) \to [0,\infty]$ (just because $\mathcal{B}(I) \subseteq \mathcal{B}(\R)$) which is also translation invariant and satisfies Equation 1.19 for any interval $[a,b] \subseteq I$.

Some notable facts, to get a little bit of a feel for the guy despite our only brief discussion (for some proofs, see Exercise 1.14):

Any singleton has Lebesgue measure $0$ i.e. for any $x \in \R$ we have that $\lambda(\{x\})=0$, and in addition to Equation 1.19 we have that any type of interval with end points $a<b$ has Lebesgue measure $b-a$: \[\lambda((a,b))=\lambda([a,b))=\lambda((a,b])=b-a.\]
The Lebesgue measure is not a finite measure (cf. Definition 1.50). Indeed for instance for any $x \in \R$ we have that $\lambda((-\infty,x))=\infty$ and $\lambda((x,\infty))=\infty$.
Any countable $A \subseteq \R$ is a Borel set and has Lebesgue measure $0$ i.e. $\lambda(A)=0$. Note one maybe somewhat striking consequence of this: the rational numbers $\Q$ are dense in $\R$ (cf. Remark 1.19) and hence you would maybe expect that their Lebesgue measures/“lengths” couldn’t be too different. However since $\Q$ is countable (cf. Definition 1.16) we have that $\lambda(\Q)=0$ while $\lambda(\R)=\infty$ (use for instance property ii above together with Theorem 1.51 i)!
And one more that maybe irks your initual intuition a little bit (at least I remember that was the case for me when I first encountered this as student): there do also exist uncountable Borel sets with Lebesgue measure $0$ (a particularly famous example is the Cantor set)!

Remark 1.57. Actually, the “natural” domain of $\lambda$ is slightly larger than the Borel $\sigma$-algebra, namely a $\sigma$-algebra whose elements are (appropriately!) called “Lebesgue measurable sets”. It is still true that not every subset of $\R$ is Lebesgue measurable, after all the Vitali sets mentioned earlier in this subsection still destroy that dream. Within pure measure theory, this slightly larger $\sigma$-algebra is slightly more convenient to use. However if the focus is on Prob theory, like ours is, sticking with the Borel $\sigma$-algebra has certain advantages. That’s why we chose the formulation we did.

Example 1.58 (Using the Lebesgue measure in a probability context) Note that the measure space $(\R,\mathcal{B}(\R),\lambda)$ we constructed in Proposition 1.56 is not a probability space, after all $\lambda(\R)=\infty$ (cf. Proposition 1.56 ii) rather than $1$. However, if we were to take an interval $I \subseteq \R$ of length $1$ as our universe rather than $\R$ we do get a probability space!

The standard choice is then to take the interval $[0,1]$ as universe: Proposition 1.56 tells us that $([0,1],\mathcal{B}([0,1]),\lambda)$ is a measure space, and since $\lambda([0,1])=1-0=1$ (cf. Equation 1.19) it is then also a probablity space (cf. the second bullet point in Definition 1.50). And that of course means that we can throw our probability interpretation at it (cf. Definition 1.30): the universe $\Omega=[0,1]$ is the set of possible outcomes of some experiment, the elements of $\mathcal{B}([0,1])$ are our events, and $\lambda$ is our probability measure that assigns a probability/likelihood to all our events.

An interesting question then is: what kind of experiment are we dealing with here? Well since the outcome space is $[0,1]$, it is an experiment in which we select a real number from the interval $[0,1]$. Similar to the coin tossing in Example 1.52, there are (infinitely) many possible probability measures possible on the measurable space $([0,1],\mathcal{B}([0,1]))$, of which $\lambda$ is one, that each describe a certain “reality” for how that number from $[0,1]$ is selected (terminology borrowed from Example 1.52!).

For compare & contrast, here is one other possible probability measure $\P$ on $([0,1],\mathcal{B}([0,1]))$: \[\P(B)=\begin{cases} 1 & \text{if } 1/2 \in B \\ 0 & \text{if } 1/2 \not\in B \end{cases} \quad \text{for all } B \in \mathcal{B}([0,1]) \] (convince yourself that this is indeed a probability measure i.e. satisfies the conditions in Definition 1.50!). What this guy does is make any Borel set in $[0,1]$ that contains the point $1/2$ almost certain to happen and every Borel set in $[0,1]$ a null event — recall from Definition 1.55 (and observe that of course we are dealing with an uncountable universe here, cf. Example 1.17 v, and hence an advanced experiment in the terminology of Definition 1.31!). If you don’t entirely see what that exactly applies just yet, then note that in particular \[\P(\{1/2\})=1, \quad \P([0,1/2))=0, \quad \P((1/2,1])=0.\] That is, it is “almost certain” that the outcome of the experiment is the number $1/2$ and (complementary of course) “almost impossible” that the outcome of the experiment is an element of one of the intervals $[0,1/2)$ and $(1/2,1]$. Hence $\P$ fits the (trivial) “reality” that we simply always select the number $1/2$.

Ok, now back to the Lebesgue measure i.e. to $([0,1],\mathcal{B}([0,1]),\lambda)$. This is a very different situation: each possible outcome $x \in [0,1]$ of the experiment has probability \[\lambda(\{x\})=0 \tag{1.20}\] (cf. Proposition 1.56 i) of happening. (If you are wondering how this is possible, then recall that we extensively discussed this in Example 1.54, see in particular the discussion around Equation 1.13!). So what “reality” fits $\lambda$? To answer this, imagine first that we cut up the outcome space $[0,1]$ in $n$ equally long parts, say \[J_1=\left[0,\frac{1}{n} \right], \quad J_2=\left( \frac{1}{n},\frac{2}{n} \right], \quad \ldots, \quad J_n=\left( \frac{n-1}{n},1 \right],\] so that for any $k=1,\ldots,n$, \[\lambda(J_k)=\frac{k}{n}-\frac{k-1}{n}=\frac{1}{n}.\] This means that the outcome of the experiment is equally likely to come from any of $J_1, \ldots, J_n$. Now imagine that you let $n$ grow, so that you get ever more and ever shorter intervals $J_1, \ldots, J_n$. For any $n$, it remains true that the outcome of the experiment is equally likely to come from any of $J_1, \ldots, J_n$. What kind of picture does that paint? Well indeed, a bit loosely speaking/at an intuitive level at least (because of Equation 1.20!), that the “reality” of this experiment is that each $x \in [0,1]$ is “equally likely” to be selected as outcome of the experiment no?

Indeed, $([0,1],\mathcal{B}([0,1]),\lambda)$ is a natural analogue in the world of “advanced” experiments (cf. Definition 1.31) of all those “simple” experiments with equally likely outcomes you’ve done so often when you were still a baby Probabilist (your mum still has the photos, you were so cute and she was so proud!) like rolling fair dice, tossing fair coins etc.! In fact, it would be fair to consider $([0,1],\mathcal{B}([0,1]),\lambda)$ the mother of all probability spaces. If you would be sent to an uninhabitated island and could take only one probability space with you, this one would be a great choice. It can facilitate any “simple” experiment (doesn’t need to have all equally likely outcomes) via an “embedding” technique similar to the one we saw in Example 1.54 under “The advanced experiment case: constructing -algebras and probability measures”. If you need a random variable with any distribution, then it is always possible to construct it on this probability space (as we will see in Chapter 2). Plus more!

Exercises

You can now do Exercise 1.13–Exercise 1.14.

1.6.2 A characterisation of all finite measures on the real line [⛔]

We conclude this chapter with a beautiful way to characterise all possible finite measures on the measure space $(\R,\mathcal{B}(\R))$ (recall from Definition 1.41). It is mathematically very nice I think, however this chapter is already getting very long (sorry!) and, despite the elegance, the result is mainly of technical relevance for us. Therefore non-examinable!

A key role in this section is played by the collection of all intervals of the form $(-\infty,x]$ for $x \in \R$, let’s call this collection $\mathcal{C}$. Let’s first observe that this collection can be used as generator (cf. Definition 1.36) for the Borel $\sigma$-algebra (cf. Definition 1.41):

Lemma 1.59 (An alternative generator of the Borel sigma-algebra [⛔]) Define the following collection of subsets of $\R$: \[\mathcal{C}=\{ (-\infty,x] \, | \, x \in \R \}.\] Then we have that $\sigma(\mathcal{C})=\mathcal{B}(\R)$.

Proof. Note that for any $a<b$, \[(a,b]=(-\infty,b] \setminus (-\infty,a] \in \sigma(\mathcal{C})\] (cf. Proposition 1.35 ii) and hence also \[(a,b)=\bigcup_{i=1}^\infty (a,b-1/i] \in \sigma(\mathcal{C})\] (cf. Definition 1.33 iii). Since any open set in $\R$ can be written as a countable union of open intervals (cf. Lemma 2.1.9 in Stroock (1994)), it follows from Definition 1.33 iii that any open set in $\R$ is also an element of $\sigma(\mathcal{C})$. Since $\mathcal{B}(\R)$ is generated by the open sets in $\R$, we get that $\mathcal{B}(\R) \subseteq \sigma(\mathcal{C})$.

On the other hand, since any interval of the form $(-\infty,x]$ is a Borel set (cf. Exercise 1.9) we also have that $\sigma(\mathcal{C}) \subseteq \mathcal{B}(\R)$ and hence the result follows.

Now, let’s think about possible measures on $(\R,\mathcal{B}(\R))$. How would you go about investigating/categorising those? It seems like a bit of a daunting task: we have already discussed that (non-trivial) $\sigma$-algebras such as $\mathcal{B}(\R)$ are complicated beasts, not very easy to oversee/specify which subsets of $\R$ it exactly contains and which not, and in order to define a measure on it we need to specify a mapping that sends each of its elements (that we can hardly specify in the first place) to a non-negative number, while the mapping also needs to satisfy the conditions in Definition 1.50!

Hmm. We discussed previously that we can get a handle on $\sigma$-algebras (at least to some extent) by working with easier to understand collections that generate it. Like $\mathcal{B}(\R)$ is by definition generated by all open subsets in $\R$ (cf. Definition 1.41), and as Lemma 1.59 shows also by the simpler collection $\mathcal{C}$ (in general, any (non-trivial) $\sigma$-algebra has many possible generators). This line of thinking naturally leads to the following idea/question: would it maybe be possible to characterise/define a measure on the whole of $\mathcal{B}(\R)$ by initially defining it on a simpler collection that generates $\mathcal{B}(\R)$ only? The answer is yes, at least, if we consider finite measures only and as long as you choose a generating collection with some particular properties. And, the collection $\mathcal{C}$ from Lemma 1.59 exactly fits the bill here!

Here is a quick analogy to illustrate the process. Suppose that we have a box of Lego bricks in different shapes and sizes. We want to define a function that takes as input any possible building we can make with these bricks, and gives us back the weight of the building. It would be one hell of a job to sit down, put together all possible buildings you can make and weigh them (to specify our function exhaustively). Rather it would make more sense to only weigh the individual bricks and jot the results down. Then whenever we construct some building, which is a sequence of steps in which we add and sometimes take away bricks, the total weight and hence the value of my function naturally follows if at each step we keep track of the weight of the bricks that are added/removed: its value is naturally computed from the individual bricks in whatever building we fancy making.

A measure on $(\R,\mathcal{B}(\R))$ needs, by definition, to give us a value (think: weight) for any Borel set (think: any possible building we could make). However we also know that a measure should be countably additive, cf. Definition 1.50 (think: it should respect the principle that the weight of any building equals the sum of the weights of the individual bricks). So, if we can identify what the collection of bricks should be ($\mathcal{C}$ is a possible choice), then the value it assigns to any Borel set is fully determined by/can be computed using a simple rule from the values it assigns to the elements of $\mathcal{C}$!

In Lemma 1.60 we work out this idea. It shows that any finite measure on $(\R,\mathcal{B}(\R))$ is fully specified by the values it assigns to the elements of $\mathcal{C}$, and vice versa that if we choose any (suitable) values to assign to the elements of $\mathcal{C}$ then these choices extend to a finite measure on $(\R,\mathcal{B}(\R))$. In order to conveniently talk about “(suitable) values assigned to the elements of $\mathcal{C}$”, note that for every $x \in \R$ there is one corresponding element in $\mathcal{C}$ namely $(-\infty,x]$, i.e. for every $x \in \R$ we need one value, so we may very well represent these values as a function $f: \R \to [0,\infty)$, where $f(x)$ is the value we assign to $(-\infty,x]$ for all $x \in \R$!

Finally a word about the “suitable” in the above “(suitable) values assigned to the elements of $\mathcal{C}$”: a finite measure takes values in $[0,\infty)$ and hence so should $f$. Further as $x$ increases, the interval $(-\infty,x]$ gets larger, hence keeping in mind Theorem 1.51 i we should have that $f$ is non-decreasing. Further as we let $x \to -\infty$ the interval $(-\infty,x]$ collapses to the empty set and hence, by Theorem 1.51 v and Definition 1.50, we should have that $f(x) \to 0$. If we let $x \to \infty$ then $(-\infty,x]$ grows to become $\R$ and hence by Theorem 1.51 iv we should have that $f(x)$ tends to some finite limit which is the measure of $\R$.

Lemma 1.60 (A characterisation of all finite measures on the Borel sigma-algebra [⛔]) Denote by $\mathbf{F}$ the family of all functions $f: \R \to [0,\infty)$ that are right-continuous, non-decreasing and satisfy \[\lim_{x \to -\infty} f(x)=0 \quad \text{and} \quad \lim_{x \to \infty} f(x)<\infty\] (note that both limits are guaranteed to exist since $f$ is non-decreasing). Further denote by $\mathbf{M}$ the family of all finite measures on $(\R,\mathcal{B}(\R))$.

Then the mapping that sends $\mu \in \mathbf{M}$ to the function $f: \R \to [0,\infty)$ given by $f(x):=\mu((-\infty,x])$ is a bijection (i.e. a one-to-one correspondence) between $\mathbf{M}$ and $\mathbf{F}$.

Some notes:

Recall that a function is right-continuous on $\R$ if for every $x \in \R$, $f(x+h) \to f(x)$ as $h \downarrow 0$. Of course, this does not necessarily mean that $f$ is also continuous in $x$, after all the left hand limit $\lim_{h \downarrow 0} f(x-h)$ may not exist or not be equal to $f(x)$. However if such a function is also non-decreasing then it cannot get too wild, for instance it must be continuous everywhere on $\R$ except for in at most countably many points (see e.g. Wiki).
This result remains true if we work on an interval $I=(a,b)$ e.g. rather than the whole of $\R$, in the way that you would expect: take for $\mathbf{M}$ all finite measures on $(I,\mathcal{B}(I))$, for $\mathbf{F}$ all right-continuous, non-decreasing functions $f: I \to [0,\infty)$ with limit value $0$ resp. finite if $x \downarrow 0$ and $x \uparrow b$ resp., and map $\mu \in \mathbf{M}$ to $f$ given by $f(x):=\mu((a,x])$.
Following up on the previous bullet point, since on an interval $I=(a,b)$ the Lebesgue measure is finite (cf. Subsection 1.6.1), it should be covered by this lemma as well. Indeed what function does it get mapped to??
The function $f$ associated with a $\mu \in \mathbf{M}$ is called its distribution function. Though we’re in measure theory mode here rather than in Probability Theory mode, you may very well already see the parallels with our use of (almost) that word in Probability Theory — we’ll return to this later on!

Proof. Let $\mathcal{C}$ be as defined in Lemma 1.59. First we need to show that for a $\mu \in \mathbf{M}$, the function $f: \R \to [0,\infty)$ given by $f(x):=\mu((-\infty,x])$ is indeed a member of $\mathbf{F}$ i.e. satisfies the conditions set out in the lemma. We’ll leave this as an exercise for later.

Injective. If $\mu, \nu \in \mathbf{M}$ get mapped to the same function then they coincide on all elements of $\mathcal{C}$. But then necessarily $\mu=\nu$. This is due to the fact that $\mathcal{C}$ is closed under finite intersections (i.e. it is a so-called $\pi$-system), see e.g. Lemma 3.1.3 and Exercise 3.1.8 in Stroock (1994).

Surjective. The difficulty here is to show that, roughly speaking, if you specify values that you would would like your measure to have for elements in $\mathcal{C}$, that a measure on the whole of $\mathcal{B}(\R)$ actually exists respecting these values. We can’t go from $\mathcal{C}$ to the whole of $\mathcal{B}(\R)$ directly but need to use another collection of subsets as intermediate step.

Consider the collection $\mathcal{A}$ of all subsets $A \subseteq \R$ that can be written as \[A=I_1 \cup I_2 \cup \ldots \cup I_n \quad \text{for some $n \geq 1$}, \tag{1.21}\] where each $I_i$ takes one of the following forms:

$(-\infty,a]$ for some $a \in \R$,
$(a,b]$ for some $a,b \in \R$ with $a<b$,
or $(b,\infty)$ for some $b \in \R \cup \{-\infty\}$.

We also include $\emptyset$ in $\mathcal{A}$ (the case $n=0$ if you will). Note that for any pair of such intervals that are not disjoint, their union can be simplified to a single interval of one of these three forms, so we can assume without loss of generality that the $I_i$’s used in Equation 1.21 are disjoint.

This collection has the special property that it is closed under complements as well as finite unions (for the latter, just convince yourself that the union of two elements from $\mathcal{A}$ is again in $\mathcal{A}$) i.e. it is a so-called algebra.

Note that $\mathcal{C} \subseteq \mathcal{A}$ and hence $\sigma(\mathcal{C}) \subseteq \sigma(\mathcal{A})$. Using Lemma 1.59 we get that $\mathcal{B}(\R) \subseteq \sigma(\mathcal{A})$. On the other hand, any interval is a Borel set (cf. Exercise 1.9), hence so is any element of $\mathcal{A}$ (cf. Definition 1.33 iii), and therefore $\sigma(\mathcal{A}) \subseteq \mathcal{B}(\R)$. Altogether: $\sigma(\mathcal{A})=\mathcal{B}(\R)$.

Given some $f \in \mathbf{F}$, define the mapping $\mu: \mathcal{A} \to [0,\infty)$ by first setting \[\begin{aligned} & \mu(\emptyset)=0, \quad \mu((-\infty,a])=f(a), \\ & \mu((a,b])=f(b)-f(a) \quad \text{and} \quad \mu((b,\infty))=f(\infty)-f(b), \end{aligned} \] where $f(\infty):=\lim_{x \to \infty} f(x)$ (note that all of these are non-negative since $f$ is non-decreasing), and then for any $A=I_1 \cup I_2 \cup \ldots \cup I_n \in \mathcal{A}$ (where, recall, we assume that the $I_j$’s are disjoint) we set \[\mu(A)=\sum_{j=1}^n \mu(I_j). \tag{1.22}\]

We need to prove two properties of $\mu$ before we can make the final step. The first property is finite additivity (i.e. as in Definition 1.33 but for only finitely many disjoint elements of $\mathcal{A}$). For this it is enough to show that for disjoint $A_1, A_2 \in \mathcal{A}$ we have that \[\mu(A_1 \cup A_2)=\mu(A_1)+\mu(A_2), \tag{1.23}\] then finite additivity follows by mathematical induction e.g. However this is pretty obvious: if two sets of the form Equation 1.21 are disjoint, then all the intervals used to express them must be mutually disjoint, so that Equation 1.23 follows immediately from Equation 1.22 (of course, this argument can also be used to show finite additivity directly, but the possibility to use mathematical induction here as well is nice to observe as a useful tool in other similar situations).

The second property we need to show is that if $A_1 \supseteq A_2 \supseteq ... \in \mathcal{A}$ is a “decreasing” sequence of sets that tend to the empty set i.e. \[\bigcap_{i=1}^\infty A_i = \emptyset, \tag{1.24}\] then $\mu(A_i) \to 0$ as $i \to \infty$. For this, first assume that in the expression Equation 1.21 $A_1$ contains only intervals of the form $I_j=(a,b]$. Then in $A_2 \subseteq A_1$, each $I_j$ is either no longer present or it has become another interval of the same form, contained in $I_j$. This observation can be repeated in $A_3 \subseteq A_2$ etc.: we can follow each of the original $I_j$’s in $A_1$ separately through their own “shrinking process” as we let $i$ increase and walk through the $A_i$’s. Due to Equation 1.22, the claim follows if we can show that each $I_j$ ultimately “shrinks” to a limit set with $0$ measure.

So fix any of these $I_j$’s in $A_1$ and write it as $(a_1,b_1]$. If it vanishes i.e. becomes the empty set at any step in the process then its measure becomes $0$ at that step and there is nothing more to do. Assume now that this doesn’t happen i.e. that for each $i=2,3,\ldots$, $A_i$ contains $(a_i,b_i]$ with $a_i<b_i$, $a_i \geq a_{i-1}$ and $b_i \leq b_{i-1}$. Since the $a_i$’s and $b_i$’s each form a monotone sequence they each have a limit, say $a_\infty$ and $b_\infty$. Since $a_i<b_i$ for all $i \geq 1$, $a_\infty \leq b_\infty$. It can’t be true that $a_\infty < b_\infty$ because then the non-empty interval $(a_\infty,b_\infty]$ is a subset of each $A_i$ and hence of the intersection of all $A_i$’s, violating Equation 1.24. So we know that $a_\infty = b_\infty$. Further it also can’t happen that $a_i<a_\infty=b_\infty$ for all $i \geq 1$, because then the singleton $\{a_\infty\}$ is a subset of each $A_i$, again violating Equation 1.24.

Now, we get from the def of $\mu$ that \[\mu((a_i,b_i])=f(b_i)-f(a_i)=f(b_i)-f(b_\infty)+f(a_\infty)-f(a_i).\] We have to be slightly careful here because $f$ is not necessarily continuous (in the point $a_\infty = b_\infty$) but we do know that it is right-continuous. Hence $f(b_i)-f(b_\infty) \to 0$ as $i \to \infty$. Further as we’ve seen above, for all $i$ large enough we have $a_i=a_\infty$ and hence $f(a_\infty)-f(a_i)=0$. In conclusion, indeed $\mu((a_i,b_i]) \to 0$ as $i \to \infty$. And, reiterating the earlier observation, by our assumption that $A_1$ consists only of intervals of the form $(a,b]$ and by Equation 1.22, $\mu(A_i) \to 0$ as $i \to \infty$.

Ok – getting there! It remains to consider the cases where $A_1$ also contains an interval of the form $(-\infty,a_1]$ and/or $(b_1,\infty)$. In the former case, as we walk through the $A_i$’s in the “shrinking process”, one of the following things happen: the interval $(-\infty,a_1]$ vanishes (in which case there is nothing to show), or at some point in the process it becomes one or more intervals of the form $(c,d]$ for $c>-\infty$ and $d \leq a_1$ in which case we can apply the above arguments again, or otherwise each of the $A_i$’s contains an interval of the form $(-\infty,a_i]$ with $a_i \leq a_{i+1}$ and limit say $a_\infty$. In this latter situation, in order to satisfy Equation 1.24 it must be true that $a_\infty=-\infty$, and we can derive our desired conclusion that its measure vanishes from the property $\lim_{x \to -\infty} f(x)=0$. In an analogue manner you can deal with an interval of the form $(b_1,\infty)$.

Right! Now we finally have everything in place to be able to apply the majestic Caratheodory Extension Theorem (see e.g. Corollary 7.1.22 in Stroock (1994)), which guarantees that a finite measure $\mu$ exists on $\sigma(\mathcal{A})=\mathcal{B}(\R)$ which respects/is an extension of the mapping $\mu$ we defined on $\mathcal{A}$, and hence in particular it also has the property that $\mu((-\infty,x])=f(x)$ for all $x \in \R$.

As a final note: the Caratheodory Extension Theorem in fact guarantees uniqueness of the measure as well, but it is worthwhile to see the $\pi$-system argument we used to show the injectivity property as that is a much more direct and commonly used argument in similar situations.

1.7 Some exercises

About the exercises

Each exercise has a (rough) indication of its difficulty, as follows:

*	easier: can be solved by (almost) only using relevant definitions/results,
**	medium: in addition to relevant definitions/results, needs a limited amount of work/creativity,
*******	harder: in addition to relevant definitions/results, needs a larger amount of work/serious creativity,
⛔	non-examinable: either very challenging or too much out of scope — these exercises are intended for those of you who relish a bit extra!

The exam consists of mostly ** and *** level questions, with some * thrown in there for good measure.

A bit of preaching: it is an incredibly important part of the study process to try and work on the exercises as much as possible before the tutorials/without looking at the solutions, both for your understanding and to secure yourself a juicy exam mark! You will get stuck at times, that’s the name of the game for all of us — as the famous mathematician Paul Erdős used to say: a problem worthy of attack, proves its worth by fighting back. :). Use the hints given with many questions/talk to classmates/ask me and try again!

Full & detailed model solutions for the exercises will become available, below the exercises on this page, immediately after our tutorial.

Exercise 1.1 [**] Let’s fiddle a bit with unions and intersections!

Give the proofs for iv and v of Example 1.11.
Let $A_1, \ldots, A_n$ be sets in a universe $\Omega$. Show that for any $j \in \{1,\ldots,n\}$ we have that \[\bigcap_{i=1}^n A_i \subseteq A_j \quad \text{and} \quad A_j \subseteq \bigcup_{i=1}^n A_i\] (observe that it this also shows that $\cap_{i=1}^n A_i \subseteq \cup_{i=1}^n A_i$).
Can you think of an example where ii holds with the “$\subseteq$”’s replaced by “$=$”’s?

Hint (click to view)

For part i, try to follow the same arguments as we used in the other examples in Example 1.11, where you may want to use the “two step” strategy from Remark 1.9 again. For the proof of iv from Example 1.11, one of the steps is quite straightforward. For the harder step you could consider using “proof by contraposition” as also seen in Example 1.11. For the proof of v, you may want to use as part of your argument the following observation. Since $1/i \to 0$ as $i \to \infty$, we also have that $-2+1/i \downarrow -2$ and $2-1/i \uparrow 2$ as $i \to \infty$. In particular this means that if you fix any $-2<x<2$, then for all $i$ large enough it holds that $-2+1/i<x<2-1/i$.

For part ii, just carefully follow the strategy from Remark 1.9, as you did in part i, i.e. translate showing “$A \subseteq B$” into showing “if $x \in A$ then $x \in B$”.

For part iii, what happens if all the sets $A_1, \ldots, A_n$ are identical?

Full solution (click to view)

For part i, let’s start with iv from Example 1.11. We use again (as in Example 1.11) the strategy from Remark 1.9:

$\cap_{i=1}^\infty A_i \subseteq \{1\}$: we need to show that if $x \in \cap_{i=1}^\infty A_i$, then $x \in \{1\}$. Using “proof by contraposition” (cf. Example 1.11) we may equivalently show that if $x \not\in \{1\}$, then $x \not\in \cap_{i=1}^\infty A_i$. So let $x$ be real number unequal to $1$. If $x<1$ then clearly $x$ is not an element of any of the $A_i$’s and hence neither of $\cap_{i=1}^\infty A_i$ (by def, cf. Equation 1.5). If $x>1$, then for all $i$ large enough (in particular $i \geq 1/(x-1)$) we have that $x \not\in [1,1+1/i)=A_i$ and hence also $x \not\in \cap_{i=1}^\infty A_i$ (again by def, cf. Equation 1.5).
$\{1\} \subseteq \cap_{i=1}^\infty A_i$: let $x \in \{1\}$ i.e. $x=1$. Then clearly for any $i=1,2,\ldots$, $x \in [1,1+1/i)=A_i$. Hence by def (cf. Equation 1.5) also $x \in \cap_{i=1}^\infty A_i$.

Next let’s do v from Example 1.11 in the same way:

$\cup_{i=1}^\infty A_i \subseteq (-2,2)$: we need to show that if $x \in \cup_{i=1}^\infty A_i$, then $x \in (-2,2)$. So let $x \in \cup_{i=1}^\infty A_i$. Then by def (cf. Equation 1.4) there exists an $i \in \{1,2,\ldots\}$ so that $x \in A_i=[-2+1/i,2-1/i]$. Clearly this means that $-2+1/i \leq x \leq 2-1/i$ which implies that $-2<x<2$ i.e. that $x \in (-2,2)$.
$(-2,2) \subseteq \cup_{i=1}^\infty A_i$: now we need to show that if $x \in (-2,2)$, then $x \in \cup_{i=1}^\infty A_i$. So let $x \in (-2,2)$. Using the def (cf. Equation 1.4), we want to show that there exists an $i \in \{1,2,\ldots\}$ so that $x \in A_i=[-2+1/i,2-1/i]$ i.e. $-2+1/i \leq x \leq 2+1/i$. It would be perfectly fine to argue here as follows: since $1/i \to 0$ as $i \to \infty$, we also have that $-2+1/i \downarrow -2$ and $2-1/i \uparrow 2$ as $i \to \infty$. Since $x$ is strictly between $-2$ and $2$ (!!), there must exist an $i$ large enough so that $-2+1/i \leq x \leq 2+1/i$, and we are done.

If you would rather prefer to explicitly specify the values of $i$ that make this work, like we have done above and in the other examples in Example 1.11, you could e.g. do this as follows. To have $-2+1/i \leq x \leq 2+1/i$, we need $x \geq -2+1/i$ and $x \leq 2+1/i$, which we can rewrite to $i \geq 1/(x+2)$ and $i \geq 1/(x-2)$ (where we used that $x+2>0$ and $x-2<0$). Since these both have to be true, we can express this as \[i \geq \max\{1/(x+2),1/(x-2)\}. \tag{1.25}\] Note that also in this argument it is crucially important that $x$ is strictly between $-2$ and $2$! Indeed the algebra we have just done needs this, and you can also see weird stuff popping up in Equation 1.25 otherwise: if $x=-2$ or $x=2$ we would be dividing by $0$!

For part ii, fix some $j \in \{1,\ldots,n\}$.

To show that $\cap_{i=1}^n A_i \subseteq A_j$, we need to show that if $x \in \cap_{i=1}^n A_i$, then $x \in A_j$. So let $x \in \cap_{i=1}^n A_i$. By def (cf. Equation 1.5) this means that $x \in A_1$, $x \in A_2$, …, $x \in A_n$ all hold. So also $x \in A_j$ holds, since $j \in \{1,\ldots,n\}$.
Next to show that $A_j \subseteq \cup_{i=1}^n A_i$, we need to show that if $x \in A_j$, then $x \in \cup_{i=1}^n A_i$. Let $x \in A_j$. Since $j \in \{1,\ldots,n\}$, this means that $x$ is an element of (at least) one of $A_1$, $A_2$, …, $A_n$, and hence by def (cf. Equation 1.4) also an element from $\cup_{i=1}^n A_i$.

For part iii, any situation in which the sets $A_1, \ldots, A_n$ are identical works. Indeed, if you take some set $A \subseteq \Omega$ and set $A_1:=A$, $A_2:=A$, …, $A_n :=A$, then it is clear from the definitions Equation 1.4 and Equation 1.5 that \[\bigcap_{i=1}^n A_i =A \quad \text{and} \quad \bigcup_{i=1}^n A_i=A.\] Further obviously for any $j \in \{1,\ldots,n\}$ we have that $A_j=A$ and hence indeed \[\bigcap_{i=1}^n A_i =A_j \quad \text{and} \quad \bigcup_{i=1}^n A_i=A_j.\]

Exercise 1.2 [***] Let’s discuss one fascinating (and later on quite useful) fact related to the “countable vs uncountable” discussion in Subsection 1.3.2. Let $f: \R \to \R$ be any continuous function. Prove that the following two statements are equivalent:

$f(q)=0$ for all $q \in \Q$,
$f(x)=0$ for all $x \in \R$.

What? Yes indeed — for a continuous function it is, a bit roughly speaking, enough to know what values it takes on the rational numbers! It follows, for instance, that if $f: \R \to \R$ and $g: \R \to \R$ are continuous functions, then $f(q)=g(q)$ for all $q \in \Q$ implies that $f(x)=g(x)$ for all $x \in \R$ (just apply the above to $f-g$) i.e. that $f$ and $g$ are actually the same function!

Extra challenge: if $f: \R \to \R$ is still a continuous function, do you think that the statements

$f(q)>0$ for all $q \in \Q$,
$f(x)>0$ for all $x \in \R$

are equivalent? Extra extra challenge ;): if you decide not, can you then formulate what makes the first equivalence work and the second fail??

Hint (click to view)

You may want to break the proof up in two steps: “$\Rightarrow$” and “$\Leftarrow$” (as per Remark 1.9). One of the directions is trivial. For the other, don’t forget that since $f$ is continuous, if $(x_n)_{n=0,1,\ldots}$ is any sequence of real numbers with limit $x \in \R$ then we have that $\lim_{n \to \infty} f(x_n)=f(x)$. Also don’t forget about the special relationship between $\Q$ and $\R$ that we mentioned in Remark 1.19!

For the extra challenge, the answer is indeed “no”! And don’t forget: to show that some statement is not true, a single example is enough!

Full solution (click to view)

Following the general strategy from Remark 1.9, we break the proof up in two steps:

i $\implies$ ii: we need to show that if $f(q)=0$ for all $q \in \Q$, then $f(x)=0$ for all $x \in \R$. So assume that $f(q)=0$ for all $q \in \Q$ and take any $x \in \R$. Because $\Q$ is dense in $\R$ (cf. Remark 1.19), there exists a sequence $(q_n)_{n=0,1,\ldots}$ of elements from $\Q$ so that $q_n \to x$ as $n \to \infty$. We can now see that \[f(x)=\lim_{n \to \infty} f(q_n)=\lim_{n \to \infty} 0 =0, \tag{1.26}\] where the first equation uses that $f$ is continous and the second that $f(q)=0$ for all $q \in \Q$.

ii $\implies$ i: this is trivial, since $\Q \subseteq \R$.

For the extra challenge, indeed this is not true! Here is a simple counter example. Let $a$ be any irrational number i.e. element of $\R \setminus \Q$, e.g. $a=\sqrt{2}$. Take $f: \R \to \R$ given by $f(x)=(x-a)^2$. Then $f(a)=0$, so statement ii does not hold. But $f(x)>0$ for all $x \not= a$, and since $a \not\in \Q$ statement i is true!

For the extra extra challenge: if you try to apply the same proof we used for the first equivalence to the second equivalence, then things go horribly wrong at Equation 1.26: the limit of a sequence of strictly positive numbers is not necessarily strictly positive. Indeed, consider for the instance the strictly positive numbers $a_n=1/n$ for all $n=1,2,\ldots$ which has limit $0$! In general, weak inequalities (i.e. $\leq$ and $\geq$) do stay preserved in a limit, but strong inequalities (i.e. $<$ and $>$) not necessarily.

Note that both equivalences are of the following form:

$f(q) \in B$ for all $q \in \Q$,
$f(x) \in B$ for all $x \in \R$

where in the first equivalence $B=\{0\}$ is closed while in the second equivalence $B=(0,\infty)$ is open (cf. Definition 1.13). And that is exactly where the key difference is! If $B$ is closed then the equivalence is true and the same proof as we used for the first equivalence above works, essentially because in Equation 1.26 we then have a sequence $f(q_n)$ in $B$ and the closedness of $B$ guarantees that the limit value $f(x)$ will be an element of $B$ as well (indeed that’s exactly the def of closed we saw in Definition 1.13!). If on the other hand $B$ is open though then this is no longer true and the equivalence is no longer true either. Our counter example above, for the case $B=(0,\infty)$, did the damage because we encountered a sequence of stricly positive numbers (i.e. all in $B$) with limit $0$ (no longer in $B$). :).

Exercise 1.3 [⛔] Prove the claims in Example 1.28: let $\Omega$ be some universe and $\mathcal{P}(\Omega)$ its powerset (cf. Definition 1.27), then show that:

if $\Omega$ has $n$ elements, then $\mathcal{P}(\Omega)$ has $2^n$ elements,
if $\Omega$ has infinitely many elements, then $\mathcal{P}(\Omega)$ is uncountable.

Hint: you may use the following technical fact: any uncountable set has a countable subset.

Hint (click to view)

First consider the case that $\Omega$ is either finite or countable. The key insight is to realise that there is a connection between subsets of $\Omega$ and sequences consisting of zeros and ones, by observing that creating a subset of $\Omega$ can be seen as walking through all elements of $\Omega$ and assigning each of them either a zero if they do not get included in your subset or a one if they do get included. This insight gives you a lovely relationship between the number of elements (loosely speaking) in $\mathcal{P}(\Omega)$ and the number of elements in $\{0,1\}^n$ or $\{0,1\}^\infty$ (notation here as in Definition 1.12). From here onwards, all you otherwise need is to make good use of what we have already established in Example 1.17.

For the case that $\Omega$ is uncountable, use the hint plus the work you have done above to create a lowerbound for the number of elements in $\mathcal{P}(\Omega)$.

Full solution (click to view)

In case ii, we may assume that $\Omega$ countable. This is sufficient, because if $\Omega$ is uncountable, then take a countable subset $\Omega' \subseteq \Omega$ (which is guaranteed to exist by the hint) as your new universe, use ii to deduce that $\mathcal{P}(\Omega')$ is uncountable, and since $\mathcal{P}(\Omega') \subseteq \mathcal{P}(\Omega)$ also $\mathcal{P}(\Omega)$ is uncountable (cf. Example 1.17 ii).

So we may write $\Omega=\{\omega_1,\omega_2,\ldots,\omega_n\}$ (case i) or $\Omega=\{\omega_1,\omega_2,\ldots\}$ (case ii). Here is the key observation. To create a subset of $\Omega$ is the same as walking through all elements of $\Omega$ and deciding for each of them whether or not we want them in our subset. Let’s flag the element with a $0$ if we don’t want it, and a $1$ if we do. This means that each possible subset of $\Omega$ can be associated with a sequence containing zeros and ones, in case i a sequence of length $n$ and in case ii a sequence of infinite length. For instance, in case i the sequence $(0,0,\ldots,0)$ corresponds to the empty set, the sequence $(1,1,\ldots,1)$ to the whole of $\Omega$, the sequence $(1,0,\ldots,0)$ to the subset $\{\omega_1\}$ etc. Analogue for case ii.

In fact, this relationship creates a bijection i.e. a one-to-one correspondence between all possible possible subsets of $\Omega$ on the one hand and $A^n$ (case i) or $A^\infty$ (case ii), where $A=\{0,1\}$ and we use the power notation from Definition 1.12. Hence, Example 1.17 iii tells us that the number of elements (loosely speaking) in $\mathcal{P}(\Omega)$ is equal to the number of elements in $A^n$ (case i) resp. $A^\infty$ (case ii).

Now it is easy to see that in case i, the number of elements in $A^n$ is $2^n$, and from Example 1.17 vii we immediately get that in case ii $A^\infty$ is uncountable.

Note: this is a textbook case of what we were talking about in Remark 1.15: we have an argument in mind (the association with these sequences) but to actually carry out that idea we need to be able to “walk through” $\Omega$ (otherwise it’s not clear how we should make that association work!). Hence our beautiful idea can only be executed if $\Omega$ is countable, and therefore we dealt with the case that $\Omega$ is uncountable separately!

Exercise 1.4 [*] Consider a universe consisting of five elements, say $\Omega=\{x_1,\ldots,x_5\}$. Consider the following collection of subsets of $\Omega$: $\mathcal{F}=\big\{ \emptyset, \{x_2,x_3\},\{x_1,x_4,x_5\},\Omega\big\}$. Is $\mathcal{F}$ a $\sigma$-algebra? What about the collection $\mathcal{G}=\big\{ \emptyset, \{x_2,x_3\},x_1,x_4,x_5,\Omega \big\}$?

Hint (click to view)

Can you recognise $\mathcal{F}$ as one of the examples discussed in Example 1.34? For $\mathcal{G}$, carefully check the requirements according to Definition 1.33, which requirement does it immediately fail?

Full solution (click to view)

For $\mathcal{F}$, this is just the third example in Example 1.34 with $B=\{x_2,x_3\}$, so indeed a $\sigma$-algebra! For $\mathcal{G}$, note that this is not a collection of subsets of $\Omega$, after all it contains the elements $x_1$, $x_4$ and $x_5$ which are not subsets of $\Omega$. Therefore by very def (cf. Definition 1.33) it cannot be a $\sigma$-algebra!

Exercise 1.5 [*/**] Let $\Omega$ be any universe and $\mathcal{F}$ a $\sigma$-algebra on $\Omega$. Fix an $A \in \mathcal{F}$, and let $\mathcal{G}$ be the same collection as $\mathcal{F}$ except that it doesn’t contain $A$. Is $\mathcal{G}$ also a $\sigma$-algebra?

Hint (click to view)

The question is whether $\mathcal{G}$ satisfies all conditions in Definition 1.33. In particular, does it satisfy condition ii in that def??

Full solution (click to view)

Well no it isn’t! Since $A \in \mathcal{F}$, also $A^c \in \mathcal{F}$ by Definition 1.33 ii. So the collection $\mathcal{G}$ does contain $A^c$ but not $A$. But this means that $\mathcal{G}$ does not satisfy condition Definition 1.33 ii: $A^c \in \mathcal{G}$ but $(A^c)^c=A \not\in \mathcal{G}$, and hence it is not a $\sigma$-algebra.

Exercise 1.6 [**/***] Let $\mathcal{F}$ be a $\sigma$-algebra on some universe $\Omega$. Fix some $B \in \mathcal{F}$. Let $\mathcal{G}=\{ A \cap B \, | \, A \in \mathcal{F} \}$, i.e. the collection of all subsets of $\Omega$ that are of the form $A \cap B$ for some $A \in \mathcal{F}$. Prove that $\mathcal{G}$ is a $\sigma$-algebra on the universe $B$.

Some notes:

What happens here is that you effectively “zoom in” from your original universe $\Omega$ to the smaller subset $B$ as your new universe. You create $\mathcal{G}$ on $B$ by going through all elements $A$ in $\mathcal{F}$ and only keeping the part of $A$ that lies in $B$ (if any). The result is a collections of subsets that are all contained in your new universe $B$, and as this exercise shows it even conveniently inherits the $\sigma$-algebra properties from $\mathcal{F}$! This construction is visually illustrated in Figure 1.2.
It is important to appreciate that we consider $\mathcal{G}$ in the context of the universe $B$. For example, in this context the complement of a set $A \in \mathcal{G}$ is the set $\{ \omega \in B \, | \, \omega \not\in A\}$.

Hint (click to view)

We will need to show that the collection $\mathcal{G}$ satisfies all three conditions from Definition 1.33. This is not overly difficult but maybe a bit fiddly.

For property ii, let $A \in \mathcal{G}$. By def of $\mathcal{G}$, this means that $A=C \cap B$ for some $C \in \mathcal{F}$. Now try to express the complement of $A$ (in the context of the universe $B$, see the second bullet point above) in the form $\ldots \cap B$, and then show that the $\ldots$ bit is an element of $\mathcal{F}$. For this, since $\mathcal{F}$ is a $\sigma$-algebra, it satisfies both the conditions in Definition 1.33 as well as the properties in Proposition 1.35!

For proprty iii, follow the same general idea. Here you may also want to use that the distributivity property also holds for infinite unions!

Full solution (click to view)

We need to check that the collection $\mathcal{G}$ satisfies all three conditions from Definition 1.33, which we can do as follows:

Since $\mathcal{F}$ is a $\sigma$-algebra, we know that $\emptyset \in \mathcal{F}$ (by Definition 1.33 i). Hence by def of $\mathcal{G}$, $\emptyset \cap B \in \mathcal{G}$. Of course $\emptyset \cap B=\emptyset$ so indeed $\emptyset \in \mathcal{G}$.
Pick any $A \in \mathcal{G}$. This means that we can write $A=C \cap B$ for some $C \in \mathcal{F}$, and (hence) in particular $A \subseteq B$. Now note that the complement of $A$ in the universe $B$ (recall the second bullet point above) can e.g. be expressed as \[\{ \omega \in B \, | \, \omega \not\in A\}=B \cap A^c=B \cap (C \cap B)^c.\] In order to show that this guy is in $\mathcal{G}$, by def of $\mathcal{G}$ we need to show that $(C \cap B)^c \in \mathcal{F}$. But we know that $B$ and $C$ are both in $\mathcal{F}$, hence so is their intersection (by Proposition 1.35 i) and hence also the complement of that intersection (by Definition 1.33 i).
Finally, let $A_1, A_2, \ldots \in \mathcal{G}$, which by def of $\mathcal{G}$ means that there are $C_1, C_2, \ldots \in \mathcal{F}$ so that $A_i=C_i \cap B$ for all $i \geq 1$. Now we can e.g. write \[\bigcup_{i=1}^\infty A_i = \bigcup_{i=1}^\infty (C_i \cap B) = \left( \bigcup_{i=1}^\infty C_i \right) \cap B\] (the final step by the distributivity property). So in order to show that this union of $A_i$’s is in $\mathcal{G}$, by def of $\mathcal{G}$ we need to show that the union of the $C_i$’s is in $\mathcal{F}$. But this holds since $C_1, C_2, \ldots \in \mathcal{F}$ and by Definition 1.33 iii.

Exercise 1.7 [*] Let $\mathcal{F}$ be a $\sigma$-algebra on some universe $\Omega$. What is $\sigma(\mathcal{F})$?

Hint (click to view)

Obviously we need to look at Definition 1.36. Hmm… Could it be… $\mathcal{F}$ itself??

Full solution (click to view)

Recall from Definition 1.36 that for any collection of subsets $\mathcal{C}$, $\sigma(\mathcal{C})$ is the smallest $\sigma$-algebra containing all the elements that are present in $\mathcal{C}$. Well, but $\mathcal{F}$ itself is a $\sigma$-algebra which trivially contains every element also present in $\mathcal{F}$ no? And clearly there is no smaller $\sigma$-algebra that also contains all the elements of $\mathcal{F}$ than $\mathcal{F}$ itself? So indeed: $\sigma(\mathcal{F})=\mathcal{F}$.

Exercise 1.8 [*/**] Let $\Omega$ be some universe, and let $\mathcal{C}_1$, $\mathcal{C}_2$ be two collections of subsets of $\Omega$ with $\mathcal{C}_1 \subset \mathcal{C}_2$ (i.e. every element of $\mathcal{C}_1$ is also an element of $\mathcal{C}_2$, but $\mathcal{C}_2$ contains strictly more elements than $\mathcal{C}_1$). Which of the following statement(s) are guaranteed to be true? And which one(s) guaranteed to be false?

$\sigma(\mathcal{C}_1) \subset \sigma(\mathcal{C}_2)$,
$\sigma(\mathcal{C}_1) \subseteq \sigma(\mathcal{C}_2)$,
$\sigma(\mathcal{C}_2) \subset \sigma(\mathcal{C}_1)$,
$\sigma(\mathcal{C}_2) \subseteq \sigma(\mathcal{C}_1)$.

Hint (click to view)

A little bit of a puzzle maybe this one, where all you need is the def from Definition 1.36. If you get stuck, then the key question to wonder is this: the “extra” elements of $\mathcal{C}_2$ i.e. those that are not in $\mathcal{C}_1$, what happens if these happen to be all elements of $\sigma(\mathcal{C}_1)$? And what happens otherwise?

Full solution (click to view)

According to Definition 1.36, $\sigma(\mathcal{C}_1)$ is the smallest $\sigma$-algebra that contains all the elements of $\mathcal{C}_1$ and similarly $\sigma(\mathcal{C}_2)$ the smallest $\sigma$-algebra that contains all the elements of $\mathcal{C}_2$.

Now, since $\mathcal{C}_1 \subset \mathcal{C}_2$, the $\sigma$-algebra $\sigma(\mathcal{C}_2)$ contains all the elements of $\mathcal{C}_1$ and hence it is either equal to $\sigma(\mathcal{C}_1)$ or it is larger than $\sigma(\mathcal{C}_2)$. So, the inclusion $\sigma(\mathcal{C}_1) \subseteq \sigma(\mathcal{C}_2)$ is guaranteed to hold.

The inclusion $\sigma(\mathcal{C}_1) \subset \sigma(\mathcal{C}_2)$ is not guaranteed to hold, it may or may not hold. Indeed, if all the “extra” elements in $\mathcal{C}_2$, i.e. those in $\mathcal{C}_2 \setminus \mathcal{C}_1$ are elements of $\sigma(\mathcal{C}_1)$ then $\sigma(\mathcal{C}_1)=\sigma(\mathcal{C}_2)$. Otherwise if that is not the case, then $\sigma(\mathcal{C}_1) \subset \sigma(\mathcal{C}_2)$.

This also shows that $\sigma(\mathcal{C}_2) \subseteq \sigma(\mathcal{C}_1)$ may or may not hold (it holds if and only if $\sigma(\mathcal{C}_1)=\sigma(\mathcal{C}_2)$), while $\sigma(\mathcal{C}_2) \subset \sigma(\mathcal{C}_1)$ is guaranteed to be false.

Exercise 1.9 [*/**] Recall from Subsection 1.5.3 that the Borel $\sigma$-algebra $\mathcal{B}(\R)$ is generated by all open subsets of $\R$. It then also contains all closed subsets (by Definition 1.33 ii). And so in particular it also contains all open intervals $(a,b)$ and all closed intervals $[a,b]$.

Now show that the following are also Borel sets i.e. elements of $\mathcal{B}(\R)$:

any singleton $\{x\}$ for $x \in \R$,
any countable subset of $\R$,
for any $a, b \in \R$ with $a<b$, the intervals $(a,b]$ and $[a,b)$,
for any $x \in \R$, the intervals $(-\infty,x]$, $(-\infty,x)$, $(x,\infty)$, $[x,\infty)$.

Hint (click to view)

The essential strategy is to express each of these sets as the complement or a (countable) union/intersection of sets that are given/you know to be Borel, so that you can apply Definition 1.33 or Proposition 1.35. And for a number of them it is even immediately clear!

If you’re struggling with part ii, then recall from Definition 1.16 that a countable subset can be expressed by listing its elements, and use this to express the subset as a union of singletons.

Full solution (click to view)

There are quite a few valid ways of doing them, below my suggestions but any other (valid ;)) way is fine as well of course!

For i, you could use that $\{x\}$ is closed (for instance because its complement is open) or that a singleton is just a special case of a closed interval: \[\{x\}=\{ z \in \R \, | \, x \leq z \leq x \}=[x,x].\]

For ii, recall from Definition 1.16 that if $A \subseteq \R$ is countable, then we can list its elements: either $A=\{a_1,\ldots,a_n\}$ or $A=\{a_1,a_2,\ldots\}$. But this also means that we can trivially write $A$ as a finite or countable union of singletons: \[A=\bigcup_{i=1}^n \{a_i\} \quad \text{or} \quad A=\bigcup_{i=1}^\infty \{a_i\}.\] Since these singletons are Borel sets by part i above, so are these unions by Definition 1.33 iii (“closed under countable unions”).

For iii, use e.g. that $(a,b]=(a,b) \cup \{b\}$. The open interval is a Borel set (by def) and so is the singleton (by part i). And then so is $(a,b]$ by Definition 1.33 iii (“closed under countable unions”). Analogue for $[a,b)$.

For iv, just observe that each of these intervals is either open or closed!

Exercise 1.10 [**] Consider some universe $\Omega$ with $\sigma$-algebra $\mathcal{F}$, and let $\mu: \mathcal{F} \to [0,\infty]$ be a mapping satisyfing the countable additivity condition of Definition 1.50. Show that if an $A \in \mathcal{F}$ exists with $\mu(A)<\infty$, then it follows that $\mu(\emptyset)=0$.

Note: this shows that any mapping $\mu: \mathcal{F} \to [0,\infty]$ satisyfing countable additivity is either a measure (i.e. satisfies all conditions in Definition 1.50), or it is a trivial mapping satisfying $\mu(A)=\infty$ for all $A \in \mathcal{F}$.

Hint (click to view)

This is one of those that is very quick provided you can find the right place to start with your argument! If you can’t find that right place, then consider applying the countable additivity property to the sequence consisting of only two sets: $A$ and $\emptyset$.

Full solution (click to view)

Suppose that $A \in \mathcal{F}$ satisfies $\mu(A)<\infty$. If you’re not sure where to start with something like this, just write down the few things that you do know/are given and play a bit around with them! All we know is that $\mu$ satisfies countable additivity (cf. Definition 1.50 ii), so we’ll very likely have to do something with that right! And what are the elements of $\mathcal{F}$ that play a role in this question? Well $A$ and $\emptyset$. So a first stab could be to apply countable additivity to the sequence consisting of these two sets only (they’re clearly disjoint), which brings us: \[\mu(A \cup \emptyset) = \mu(A)+\mu(\emptyset).\] But, of course $A \cup \emptyset=A$ and hence this implies \[\mu(A) = \mu(A)+\mu(\emptyset)\] i.e. $\mu(\emptyset)=\mu(A)-\mu(A)=0$, as we were looking for!

Note that if $\mu(A)=\infty$ the above still works, but then we’d end up with $\mu(\emptyset)=\infty-\infty$ which is an undefined quantity.

Exercise 1.11 [***] Prove properties i–iv of Theorem 1.51 (you can do v as well if you like, but it is quite similar to iv).

Maybe you want to leave property iii to do last — that’s probably the trickiest one!

Hint (click to view)

Recall that from its definition, Definition 1.50, we only know how to compute the measure of the empty set and of a finite or countable union of mutually disjoint sets. So if we want to derive an expression for the measure of anything else, we’ll need to try and express that “anything else” in terms of the empty set/a finite or countable union of mutually disjoint sets. That’s the essential strategy. Recall that Venn diagrams are always handy to create a picture for yourself of what is going on and where mutually disjoint sets are lurking that you could be exploiting!

Properties i and ii can be done in a few lines using that essential strategy.

For property iv, if you can’t seem to get started with that one, then consider defining the sequence of $B_1:=A_1$, $B_j := A_j \setminus A_{j-1}$ for all $j \geq 2$ and try to show that \[A_n =\bigcup_{j=1}^n B_j \quad \text{and} \quad A =\bigcup_{j=1}^\infty B_j.\]

For property iii, one possible route is first to use mathematical induction to prove that for any $n \geq 2$ it holds that \[\mu \left( \bigcup_{i=1}^n A_i \right) \leq \sum_{i=1}^n \mu(A_i).\] Then define $B_n:=\cup_{i=1}^n A_i$ for all $n \geq 1$ and look at applying property iv that you have just proven to it!

Full solution (click to view)

Note: all sets we create in these solutions consist of unions/intersections/differences of sets that we assume to be in $\mathcal{F}$ (as part of the assumptions in Theorem 1.51), hence from Definition 1.33 & Proposition 1.35 we know that all these created sets are also in $\mathcal{F}$ and that hence they are valid arguments for the measure $\mu$. Further, where we need to deal with infinite unions/intersections, we also use the observations we made in Definition 1.10 and the kind of reasoning we used in Example 1.11 can be quite handy as well!

We do properties i and ii first:

For property i, since $A \subseteq B$, we can write $B$ as the union of two disjoint sets as follows: $B=A \cup (B \setminus A)$ (use Venn diagram e.g.). From Definition 1.50 ii it follows that $\mu(B)=\mu(A)+\mu(B \setminus A)$. Since $\mu(B \setminus A) \geq 0$, we indeed get that $\mu(B) \geq \mu(A)$. We also get that $\mu(B \setminus A)=\mu(B)-\mu(A)$ provided that $\mu(A)<\infty$ (otherwise we would end up with the undefined $\infty-\infty$ in the right hand side).

For property ii, looking to write $A \cup B$ as the union of mutually disjoint sets, we can e.g. do that as $A \cup B=A \cup (B \setminus (A \cap B))$ (use Venn diagram e.g.). So, provided that $\mu(A \cap B)<\infty$: \[\begin{align*} \mu(A \cup B) &= \mu(A \cup (B \setminus (A \cap B))) \\ &= \mu(A)+\mu(B \setminus (A \cap B)) \\ &= \mu(A)+\mu(B)-\mu(A \cap B), \end{align*}\] where the second equality uses Definition 1.50 ii and the third uses the result of part i above (indeed $A \cap B \subseteq B$).

For property iv, recall from Definition 1.50 that essentially the only real tool we have for dealing with a sequence of subsets $A_1, A_2, \ldots$ is countable additivity, but this requires the subsets to be mutually disjoint which is clearly not the case for the non-decreasing sequence of events we’re dealing with here. So we need to figure out a way to bring mutually disjoint sets into play! The key observation is that we may write for any $n \geq 1$ \[\begin{align*} A_n &= A_1 \cup A_2 \setminus A_1 \cup A_3 \setminus A_2 \cup \ldots \cup A_n \setminus A_{n-1} \\ &= \bigcup_{j=1}^n B_j, \end{align*} \tag{1.27}\] where we define $B_1:=A_1$, $B_j := A_j \setminus A_{j-1}$ for all $j \geq 2$. If you don’t see what’s going on here, draw a pic of a growing sequence of sets $A_1$, $A_2$, $A_3$ and see where these set differences fit into your pic. The beauty of this is that the sequence $B_1, B_2, \ldots$ is mutually disjoint, and hence by countable additivity (cf. Definition 1.50 ii) it follows that for any $n \geq 1$ \[\mu(A_n)=\sum_{j=1}^n \mu(B_j). \tag{1.28}\] Further we claim that we may write the limit set $A$ as the union of the $B_j$’s: \[A=\bigcup_{j=1}^\infty B_j. \tag{1.29}\] To see that this holds, first pick any element $\omega \in A$. By def of $A$, i.e. $A=\cup_{n=1}^\infty A_n$, there must exist an $n$ so that $\omega \in A_n$. By Equation 1.27, also $\omega \in \cup_{j=1}^n B_j$ and hence also $\omega \in \cup_{j=1}^\infty B_j$. The other way around, pick any element $\omega \in \cup_{j=1}^\infty B_j$. Then there must exist an $i$ so that $\omega \in B_i$. Again by Equation 1.27, it follows that $\omega \in A_n$ for all $n \geq i$ and hence also $\omega \in \cup_{n=1}^\infty A_n$.

Finally, having established Equation 1.29, since the $B_j$’s are mutually disjoint, we can apply countable additivity to get \[\mu(A)=\mu \left( \bigcup_{j=1}^\infty B_j \right) = \sum_{j=1}^\infty \mu(B_j)\] and using that an infinite sum is nothing but the limit of partial sums and Equation 1.28 we get exactly what we were looking for: \[\mu(A)=\lim_{n \to \infty} \sum_{j=1}^n \mu(B_j) = \lim_{n \to \infty} \mu(A_n).\] Done!

For property iii finally then! This is one of these where the thing you want to prove seems fairly obvious (after all it translates to: the “size” of a union can never be larger than the sum of the “sizes” of the components) but then it is not so clear how to actually go about it. There is a few ways to do it (and any valid way is fine of course). One way is to do something similar as we did in the proof of property iv. Another route is to observe that property ii above implies a simplified version of the claim: $\mu(A \cup B) \leq \mu(A)+\mu(B)$. Maybe we can throw some mathematical induction add this to extend it beyond two sets?

Ok, for starters, note that we can safely assume that $\mu(A_i)<\infty$ for all $i \geq 1$. After all, if that were not true then $\sum_{i=1}^\infty \mu(A_i)=\infty$ and the statement we are trying to prove is trivially true. Next let’s use induction to prove that for any $n \geq 2$ it holds that \[\mu \left( \bigcup_{i=1}^n A_i \right) \leq \sum_{i=1}^n \mu(A_i). \tag{1.30}\] For this, the “base case” with $n=2$ follows from property ii. For the “induction step”, take any $n \geq 3$ and assume that Equation 1.30 holds for $n-1$ (the “induction hypothesis”). Then we can deduce \[\begin{align*} \mu \left( \bigcup_{i=1}^n A_i \right) &= \mu \left( A_n \cup \bigcup_{i=1}^{n-1} A_i \right) \\ &\leq \mu(A_n) + \mu \left( \bigcup_{i=1}^{n-1} A_i \right) \\ &\leq \mu(A_n) + \sum_{i=1}^{n-1} \mu(A_i) \\ &= \sum_{i=1}^{n} \mu(A_i), \end{align*}\] where the first inequality uses property ii (applied to the sets $A_n$ and $\cup_{i=1}^{n-1} A_i$) and the second the “induction hypothesis”. So we have now completed the induction process and proven that Equation 1.30 indeed holds for all $n \geq 2$.

Be aware that this is still not entirely what we need to prove though. The obvious way to try and extend this to the statement we need is to take the limit for $n \to \infty$ at both sides of Equation 1.30. That gives us what we want on the right hand side, but not really on the left hand side: we get a limit in front of the $\mu$ while we actually need it inside the $\mu$. Note that you can’t just put the limit inside without any further ado because this is not always valid/true, it needs a proof that you can do that! It essentially needs a “continuity” property of $\mu$, of which we have seen two specific formulations in properties iv and v. So let’s see if we can use those. We could for instance define $B_n := \cup_{i=1}^n A_i$ for all $n \geq 1$. Then the $B_n$’s form a non-decreasing sequence (if we increase $n$ we only add more sets to the union so it definitely doesn’t get any smaller) and you can easily check that (*): \[\bigcup_{n=1}^\infty B_n = \bigcup_{i=1}^\infty A_i.\] Using this we then finally arrive at our result: \[\begin{align*} \mu \left( \bigcup_{i=1}^\infty A_i \right) &= \mu \left( \bigcup_{n=1}^\infty B_n \right) \\ &= \lim_{n \to \infty} \mu(B_n) \\ &= \lim_{n \to \infty} \mu \left( \bigcup_{i=1}^n A_i \right) \\ &\leq \lim_{n \to \infty} \sum_{i=1}^n \mu(A_i) \\ &= \sum_{i=1}^\infty \mu(A_i), \end{align*}\] where the second equality uses property iv, the third the definition of $B_n$, and the inequality uses Equation 1.30. Done!

(*): To show that the union of $B_n$’s is a subset of the union of the $A_i$’s, if $x$ is an element of the union of the $B_n$’s then it must be an element of $B_n$ for some $n$. By def of $B_n$ it is then also an element of $\cup_{i=1}^n A_i$, and so it must be an element of one of $A_1, \ldots, A_n$. But then it also an element of the union of all $A_i$’s. You can go the other way by yourself!

Exercise 1.12 [**/***] Let $(\Omega,\mathcal{F},\P)$ be some probability space.

Show that $A$ is an almost sure event if and only if $A^c$ is a null event.
Suppose that $A_1, A_2, \ldots$ is a (countable) sequence of almost sure events. Show that $\cap_{i=1}^\infty A_i$ is also an almost sure event.

Hint: note that to prove a statement like “$B$ is an almost sure event” you need to do two things: first show that $B$ is indeed an event i.e. that $B \in \mathcal{F}$, and then show that $\P(B)=1$ (cf. Definition 1.55).

Note 1: at this point, part ii may seem like a bit of a soulless technical result. However, as we go forward it’ll regularly turn out to be an extremely valuable help result!

Note 2: note that it follows from part ii that each of $\cap_{i=1}^n A_i$, $\cup_{i=1}^n A_i$ and $\cup_{i=1}^\infty A_i$ are also all almost sure events, as each of them has $\bigcap_{i=1}^\infty A_i$ as a subset (this is partially shown in Exercise 1.1, the arguments used there can be slightly extended to show this as well).

Hint (click to view)

Of course, for any question the starting point is to be fully aware of (and otherwise look up!) all the context/ingredients relevant for the question. In this case, recall that a probability space is discussed in Definition 1.50: the events are all the elements of the $\sigma$-algebra $\mathcal{F}$, and $\P$ is a probability measure i.e. a measure as discussed in Definition 1.50 and Theorem 1.51, with the additional property that $\P(\Omega)=1$. Further recall from Definition 1.55 that an event $A$ is called an almost sure event resp. a null event if $\P(A)=1$ resp. $\P(A)=0$.

For part i, recall from Definition 1.2 iv that one (usually best) way to prove an equivalence is to break it up in two implications and prove each of those. Further, to cover the “is an event” part of your proof the defining properties of a $\sigma$-algebra are all you need (cf. Definition 1.33). To cover the probablities part, you would be very right to be looking for what you probably know from your earlier Prob courses as the “complement rule” i.e. $\P(A^c)=1-\P(A)$, but sadly it doesn’t appear anywhere in the properties we have listed! Not to worry though, it is easily derived using countable additivity (cf. Definition 1.50 ii).

For part ii, bit fiddly maybe! The “is an event” part is again easily enough resolved, but the probability part is a bit less obvious maybe: looking through Definition 1.50 and Theorem 1.51 there is unfortunately very little about intersections… There is plenty about unions though! How do we relate unions and intersections again? Indeed, the De Morgan’s laws (cf. Definition 1.10)! These are the key ingredients, with those it’s a few lines of algebra to puzzle it all together. If you can’t seem to make it work, then don’t forget about the subadditivity property (cf. Theorem 1.51 iii)!

Full solution (click to view)

Recall that the context/ingredients of a probability space are discussed in Definition 1.50: the events are all the elements of the $\sigma$-algebra $\mathcal{F}$, and $\P$ is a probability measure i.e. a measure with the additional property that $\P(\Omega)=1$. Further recall from Definition 1.55 that an event $A$ is called an almost sure event resp. a null event if $\P(A)=1$ resp. $\P(A)=0$.

For part i, following the suggestion from Definition 1.2 iv, we prove both implications separately.

For “$\implies$”, assume that $A$ is an almost sure event i.e. $A \in \mathcal{F}$ with $\P(A)=1$. Then $A^c$ is also an event, by Definition 1.33 ii. Further, we haven’t explicitly listed the “complement rule” in one of our results, but it is very easy to derive: since the events $A, A^c$ are mutually disjoint and $A \cup A^c=\Omega$, we get from countable additivity (cf. Definition 1.50 ii) that \[1=\P(\Omega)=\P(A \cup A^c)=\P(A)+\P(A^c)\] and hence \[\P(A^c)=1-\P(A).\] Since $\P(A)=1$, it indeed follows that $\P(A^c)=0$.

For “$\impliedby$” you can use exactly the same arguments, just recall that $(A^c)^c=A$.

For part ii, let’s firstly convince ourselves that $\cap_{i=1}^\infty A_i$ is indeed an event i.e. that $\cap_{i=1}^\infty A_i \in \mathcal{F}$. For this we don’t have to look very far, we immediately get it from Proposition 1.35.

So it remains to prove that $\P(\cap_{i=1}^\infty A_i)=1$. For this, as the properties we have listed for (probability) measures focus a lot more on unions than intersections, it is tempting to see if we could find a way to work with unions rather. For that we always have De Morgan’s laws (cf. Definition 1.10) of course! That gives \[\left( \bigcap_{i=1}^\infty A_i \right)^c = \bigcup_{i=1}^\infty A_i^c. \tag{1.31}\] Now trying to manipulate the probability of the right hand side, looking through the properties for (probability) measures in Definition 1.50 and Theorem 1.51 to find a suitable one… Note that we have no reason to assume that the $A_1^c, A_2^c, \ldots$ are mutually disjoint, so our loyal servant “countable additivity” is of not much use here. However there is his slightly more shy cousin who goes by the name “subadditivity” (cf. Theorem 1.51 iii)! That one yields, together with $\P(A_i^c)=0$ (cf. part i) \[\P \left( \bigcup_{i=1}^\infty A_i^c \right) \leq \sum_{i=0}^\infty \P(A_i^c)=0. \tag{1.32}\] So indeed: \[\begin{align*} \P \left( \bigcap_{i=1}^\infty A_i \right) &= 1- \P \left( \left( \bigcap_{i=1}^\infty A_i \right)^c \right) \\ &= 1- \P \left( \bigcup_{i=1}^\infty A_i^c \right) \\ &\geq 1-0 \\ &=1, \end{align*} \] where the first equality uses part i again, the second Equation 1.31, and the inequality uses Equation 1.32.

Exercise 1.13 [*] For each of the following three Borel sets, compute its Lebesgue measure (cf. Proposition 1.56).

$A=[5,10)$,
$A=[1,2] \cup (3,4)$,
the set $A$ consisting of all intervals of the form $(k,k+r^k)$ for all $k=1,2,\ldots$, for some $r \in (0,1)$.

Hint (click to view)

Besides the specific properties listed in Proposition 1.56, a Lebesgue measure, being a measure, of course also satisfies the conditions from Definition 1.50 and properties from Theorem 1.51. That’s more than plenty for this not too hard question, you only need a few of all those! Just maybe for property iii, don’t forget about your good old friend the geometric series!

Full solution (click to view)

Directly from Equation 1.19 we get that $\lambda([5,10))=10-5=5$.
Since $A$ is given as the union of two disjoint intervals, by countable additivity (cf. Definition 1.50 ii) we can write \[\lambda([1,2] \cup (3,4))=\lambda([1,2])+\lambda(3,4))\] and using Equation 1.19 as well as Proposition 1.56 i this brings us to \[\lambda([1,2] \cup (3,4))=2-1+4-3=2.\]
Note that can write \[A=\bigcup_{k=1}^\infty (k,k+r^k)\] and since $r^k \leq 1$ for all $k=1,2,\ldots$, this is a union of mutually disjoint intervals. Again by countable addivitivy (cf. Definition 1.50 ii) and Proposition 1.56 i we find \[\begin{align*} \lambda \left( \bigcup_{k=1}^\infty (k,k+r^k) \right) &= \sum_{k=1}^\infty \lambda((k,k+r^k)) \\ &= \sum_{k=1}^\infty k+r^k-k \\ &= \sum_{k=1}^\infty r^k \\ &= \frac{r}{1-r}, \end{align*}\] the final step by the good old geometric series.

Exercise 1.14 [**] Prove properties i–iii in Proposition 1.56.

Hint (click to view)

Always good to have your available tools nicely lined up when attacking a question! In this case, we know that Equation 1.19 holds, as well as the general properties of measures we’ve seen in Definition 1.50 and Theorem 1.51.

For property i, if you think about it, then a singleton is just a special case of a closed interval! Further all you need is countable additivity (cf. Definition 1.50 ii) and Equation 1.19.

For property ii, recall that the larger a set/interval, the larger its measure (cf. Theorem 1.51 i). What subsets can you find in $(x,\infty)$ of which you can easily compute the Lebesgue measure?

For property iii, the key observation to use is that a countable set can be written as a countable union of its elements as singletons, as we also used in Exercise 1.9 ii for instance!

Full solution (click to view)

Our arsenal of key tools here are that the Lebesgue measure satisfies Equation 1.19, and of course, being a measure, it also satisfies the conditions from Definition 1.50 and properties from Theorem 1.51.

For property i, one option is to observe that a singleton is just a special case of a closed interval: \[\{x\}=\{ z \in \R \, | \, x \leq z \leq x \}=[x,x]\] and hence we can immediately apply Equation 1.19 to get that \[\lambda(\{x\})=\lambda([x,x])=x-x=0. \tag{1.33}\] Further, for any $a<b$ we can for instance write $[a,b]=\{a\} \cup (a,b) \cup \{b\}$, which is clearly a mutually disjoint union and therefore we can we can apply countable additivity (cf. Definition 1.50 ii) to deduce that \[\lambda([a,b])=\lambda(\{a\})+\lambda((a,b))+\lambda(\{b\}).\] Now apply Equation 1.19 to the left hand side, and the above Equation 1.33 to the right hand side, and we get to $\lambda((a,b))=b-a$ as we wanted. The other intervals mentioned in property i can be handled in the same way!

For property ii, to take $(x,\infty)$, you could for instance use that for any $n=1,2,\ldots$ we have that $(x,x+n) \subseteq (x,\infty)$. So it follows from Theorem 1.51 i and property i above that \[\lambda((x,\infty)) \geq \lambda((x,x+n))=x+n-x=n.\] And since this is true for any $n=1,2,\ldots$ we indeed get that $\lambda((x,\infty))=\infty$.

For property iii, recall from Definition 1.16 that any countable $A \subseteq \R$ can be expressed by listing its elements i.e. e.g. $A=\{a_1, a_2, \ldots \}$ (the finite case works in the same way). In particular this shows that we can write $A$ as a (countable!) union of its own elements i.e. \[A=\bigcup_{i=1}^\infty \{a_i\}.\] Of course, these singletons are mutually disjoint, and therefore we can apply countable additivity (cf. Definition 1.50 ii) to deduce that \[\lambda(A)=\sum_{i=1}^\infty \lambda(\{a_i\}).\] Throwing property i into the mix we find what we’re looking for: \[\lambda(A)=\sum_{i=1}^\infty 0=0.\]

(a) Some universe $\Omega$ with a collection of subsets (the purple shapes, some of which overlap i.e. have non-empty intersection) $\mathcal{F}$ (note that this collection is **not** a $\sigma$-algebra, just to avoid getting a visual with too many subsets — doesn’t affect the construction principle though!)

\(P\)	\(Q\)	\(P \implies Q\)	\(P \iff Q\)
\(x=-2\)	\(x^2=4\)	holds/true	doesn’t hold/false
\(x^2=4\)	\(x=-2\)	doesn’t hold/false	doesn’t hold/false
\(x=-2\) or \(x=2\)	\(x^2=4\)	holds/true	holds/true