The Law of Large Numbers is a fundamental theorem in probability theory that describes how the average of results from a large number of trials converges toward the expected value as the number of trials increases. In simpler terms, it tells us that when we repeat a random experiment many times, the sample average gets closer and closer to the theoretical mean we would expect from the probability distribution.
To understand this concept better, imagine flipping a fair coin, we know theoretically that the probability of getting heads is 0.5, or 50%. However, if you flip the coin just 10 times, you might get 7 heads and 3 tails, giving you a 70% heads rate. This deviation from the expected 50% is perfectly normal with such a small sample. But if you flip the coin 10,000 times, you’ll likely find that the proportion of heads is much closer to 50%, perhaps 49.8% or 50.3%. This convergence is precisely what the Law of Large Numbers guarantees.
Two Versions of the Law
There are actually two versions of this law:
- Weak Law of Large Numbers: States that the probability that the sample average deviates from the expected value by more than any small amount approaches zero as the sample size increases.
- Strong Law of Large Numbers: Goes further, stating that the sample average almost surely converges to the expected value with probability one.
Key Applications
The Law of Large Numbers is crucial across numerous fields:
- Insurance Industry: Companies rely on this principle to set premiums. While they cannot predict whether any individual person will file a claim, they can predict with remarkable accuracy what percentage of thousands of policyholders will file claims, allowing them to price their products sustainably.
- Casino Operations: Casinos operate on this principle—they might lose money to individual gamblers in the short term, but over millions of bets, the house edge ensures profitability.
- Statistical Inference and Surveys: The law justifies using sample means to estimate population parameters. When conducting surveys or experiments, researchers collect data from samples to draw conclusions about entire populations. This is why political polls become more accurate with larger sample sizes and why clinical trials need substantial numbers of participants to produce trustworthy results.
- Monte Carlo Simulations: These computational algorithms rely on repeated random sampling to obtain numerical results. They are used extensively in finance for option pricing, in physics for modeling complex systems, and in engineering for risk assessment. The Law of Large Numbers ensures that as we run more simulations, our estimated results converge to the true values.
- Quality Control and Manufacturing: Companies use this principle when implementing statistical process control. By taking regular samples from production lines, they can estimate the proportion of defective items being produced and determine whether the process is operating within acceptable parameters.
Weak Law of Large Numbers
The Weak Law of Large Numbers applies to a sequence of independent and identically distributed random variables X₁, X₂, …, Xₙ, each with expected value E[Xᵢ] = μ and finite variance Var(Xᵢ) = σ². When we calculate the sample mean f(n) = (X₁ + X₂ + … + Xₙ) / n, the law states that for any small positive value ε, the probability that f(n) deviates from μ by more than ε approaches zero as n increases: lim P(|f(n) – μ| > ε) = 0 as n→∞. In other words, the larger our sample becomes, the more confident we can be that our sample average will be close to the true expected value.
Application to Bernoulli Trials
This simulation specifically examines Bernoulli trials, which represent the simplest type of random experiment. Each trial produces one of two outcomes: success (coded as 1) with probability p, or failure (coded as 0) with probability 1 – p. Crucially, each trial is independent of all others. The relative frequency after n trials is calculated as f(n) = (number of successes) / n. According to the Law of Large Numbers, as the number of trials grows indefinitely, this relative frequency converges almost surely to the true probability p.
Mathematical Properties
Several important mathematical properties govern this convergence behavior:
Confidence Intervals: Using the normal approximation, we can construct confidence intervals. With approximately 95% probability, the relative frequency f(n) will lie within the range p ± 1.96 × √(p(1-p)/n).
Variance and Convergence Rate: The variance of the relative frequency is Var(f(n)) = p(1-p) / n, which reveals that the spread of f(n) around p decreases proportionally to 1/√n. This means that to halve the uncertainty, we need to quadruple the number of trials.
Normal Approximation: For large values of n, the Central Limit Theorem tells us that f(n) follows approximately a normal distribution with mean p and variance p(1-p)/n, expressed as f(n) ~ N(p, p(1-p)/n).
Required objective
Now it will be simulated the Law of Large Numbers (LLN) by plotting multiple (m) trajectories of (n) trials for the relative frequency f(n). Show with a histogram (possibly, a vertically oriented histogram placed on the right side of the main chart) how, as soon a n becomes large, the empirical distribution of f(n) (over all m trajectories) tends to the underlying probability p of success used in each trial.
The two panels below illustrate a Monte Carlo demonstration of the Law of Large Numbers for independent Bernoulli trials. Each simulation generated m=80 independent trajectories of cumulative relative frequency f(n) over n=2000 trials. The top plot for each case shows the 80 trajectories f(n) (relative frequency of “success” at each trial index n) together with a dashed horizontal line at the theoretical probability p. The second one is a histogram of the final relative frequencies f(n=2000) across the 80 trajectories; the dashed vertical line indicates the theoretical value p.
Two cases are presented: a fair coin (p=0.5) and a biased coin (p=0.3).
The simulation outcomes demonstrate strong agreement with theoretical predictions. With n = 2000 trials, the empirical means calculated across all m = 80 trajectories are 0.499 for p = 0.5 and 0.30007 for p = 0.3. These values are virtually identical to the true probabilities, confirming that the relative frequency serves as an unbiased estimator of the underlying success probability.
The empirical standard deviations observed across the 80 trajectories are approximately 0.01026 when p = 0.5 and 0.00865 when p = 0.3. These values align closely with the theoretical standard errors predicted by the formula √(p(1-p)/n), which yields approximately 0.01118 and 0.01025 respectively. The slight discrepancies between empirical and theoretical values are entirely expected, arising from finite-sample effects and the inherent variability in estimating the sample standard deviation from only 80 independent realizations.
To assess whether any systematic bias exists, z-scores were calculated to measure how far the sample means deviate from the true probabilities. The resulting z-scores of approximately -0.8 for p = 0.5 and 0.06 for p = 0.3 are both very close to zero, indicating no statistically significant bias in either case.
Both the visual patterns observed in the plots and these quantitative measures provide compelling confirmation of the Law of Large Numbers. The results also validate the Central Limit Theorem’s prediction that sampling variability decreases proportionally to 1/√n, as evidenced by the close correspondence between empirical and theoretical standard deviations.
Two additional cases are now presented where the number of trials has been increased to n = 5000, while the number of trajectories (m) and the success probability (p) remain unchanged from the previous simulations. This extension allows us to observe how the convergence behavior strengthens with a significantly larger sample size, providing a clearer demonstration of the Law of Large Numbers in action.
Conclusion
The four simulations (m = 80 trajectories; p = 0.5 and p = 0.3; n = 2000 and n = 5000) provide a clear empirical confirmation of the Law of Large Numbers. In all cases the running relative frequency f(n) shows large fluctuations at small n and progressively stabilizes around the theoretical probability p as n increases. The histograms of the final values f(n) concentrate tightly around p, and the measured dispersion decreases when n increases from 2000 to 5000.