Decision Making Under Uncertainty: Lessons from Practice

The following is excerpted and adapted from my private notes during my time at Predikto (2014-2018), exploring how to make complex decisions under uncertainty in technical startups.

The Nature of Uncertain Decisions

Decision-making is a game of compromise. In simple cases, where information about a decision and its various outcomes is complete and available, finding the optimal path is straightforward. But these cases are boring. Let’s explore the more interesting cases where outcomes are uncertain or completely unknown, and the best choice cannot be known upfront.

Expected Values and Their Limitations

Consider a simple binary choice:

  1. Do a thing
  2. Do not do that thing

Let’s say doing the thing gives us an 80% chance of receiving $100, and a 20% chance of losing $40. If we don’t do it, we gain and lose nothing. This is “uncertain” because while we don’t know the exact outcome, we have a model to describe it.

The traditional probabilistic approach (Pascal’s) would calculate the expected value:

E[x] = 0.80 * 100 - 0.20 * 40 = 72

Over many iterations, this suggests we should “do the thing” since $72 > $0. But real-world decisions are rarely this simple.

The St. Petersburg Paradox

To understand why expected value alone is insufficient, let’s examine the famous St. Petersburg Paradox, proposed by Daniel Bernoulli in 1738. Imagine a casino game:

  • A fair coin is tossed repeatedly
  • The pot starts at $2 and doubles every time heads appears
  • When tails appears, the game ends and you get the pot
  • Question: What’s a fair price to play?

Using pure expected value:

E = (1/2 * 2) + (1/4 * 4) + (1/8 * 8) + (1/16 * 16) + ...
  = 1 + 1 + 1 + 1 + ...
  = ∞

This suggests you should pay any finite amount to play - clearly absurd! The paradox reveals that we need to consider:

  • Finite resources
  • Risk tolerance
  • Utility (the same dollar amount means different things to different people)

Real-World Decision Trees

Real decisions are rarely binary and always have associated costs. Let’s examine a practical example: buying milk.

You have two options:

  1. Drive 10 minutes to a store you know has milk (100% certainty)
  2. Walk 5 minutes to a closer store that might have milk (80% certainty)

If the small store doesn’t have milk, you’d need to:

  1. Walk back home (5 minutes)
  2. Then drive to the certain store (10 minutes)

This creates expected costs:

E[Drive Direct] = 20 minutes (certain)
E[Try Small Store] = 14 minutes (expected)
min[Small Store] = 10 minutes
max[Small Store] = 30 minutes

But wait! What if you have a deadline? What if time has non-linear value? These factors create complex decision spaces where the “obvious” choice might not be optimal.

The Cost of Exploration in Startups

This framework has profound implications for technical startups in nascent markets. Consider the choice between:

  1. Building product in-house
  2. Using third-party tools/services

The uncertainty differs in each path:

  • In-house: Higher certainty in control/flexibility, paid for in time/cost
  • Third-party: Faster iteration but risk of having to revert

The key insight is that reversion cost increases with project maturity. A mature product is much more expensive to refactor than a prototype.

Making Decisions with Incomplete Information

Through simulation and tinkering, we can understand how different parameters affect outcomes:

  1. If 50% of work can be salvaged when reverting from third-party:

    • Stay in-house if probability of failure > 30%

  2. If only 10% can be salvaged:

    • In-house is better for all but the most certain third-party cases

  3. If 90% can be salvaged:

    • Third-party is better regardless of other factors

The relationship between these factors is highly non-linear and often counterintuitive.

A Better Way to Think About Uncertainty

Instead of starting with assumptions and projecting outcomes, work backwards:

  • Start with desired outcomes
  • Determine what assumptions would need to be true
  • Evaluate the validity of those assumptions

For example, instead of asking:

“What’s the probability the third-party vendor will fail?”

Ask:

“Can we reuse more than 50% of their work if they do fail?”

Or better yet:

“How much of the work would need to be re-usable for this to still make sense?”

The Startup Context

Technical startups in nascent markets face unique challenges:

  1. Customer desires are uncertain
  2. Technical feasibility is uncertain
  3. Task scoping is uncertain
  4. Each option has non-zero exploration cost

The real key here is to change your mindset from trying to eliminte uncertainty, or predict the future, to instead understanding the sttructure of the problems you’re working through, and back into what the certainty would have to be to move forward. This is a much easier position to decide from.

At some point, you have to make a bet. The goal isn’t to eliminate uncertainty - that’s impossible. The goal is to understand the bet you’re making and structure your organization to be resilient to different outcomes.

Note: This post adapts mathematical concepts and decision trees into a more accessible format. The original analysis included some cool graphs and simulations using a project called petersburg I had released on github. I don’t have the images from this handy, but will try to dig them up and add them here if I can.