Activating project at `~/Teaching/BEE4750/fall2023/slides`
Lecture 25
November 15, 2023
Activating project at `~/Teaching/BEE4750/fall2023/slides`
Robustness: How well a decision performs under plausible alternative specifications.
Note: We are using “robustness” in a slightly different sense than some other related uses.
If you see “robust optimization,” that is a different thing: an approach to mathematical programming under uncertainty.
Robustness: summarizing how well a decision works across a range of different plausible futures.
Robustness can be measured using a variety of metrics (more on this later).
Suppose we estimate \(q=2.5\), \(b=0.4\), and runoffs \(y_t\sim~LogNormal(0.03, 0.25).\)
After testing against 1000 runoff samples, we decide to try a release of \(a_t = 0.04\) units each year.
Now suppose \(q=2.4\) instead…
Or \(b=0.39\)…
Or \(b=0.41\)…
Given an assessment of performance over a variety of specifications (or states of the world), there are a number of metrics that can be used to capture robustness, and the choice can matter quite a bit.
Two common ones are satisfycing and regret.
Satisfycing metrics try to express the degree to which performance criteria are satisfied across the considered states of the world.
A simple satisfycing metric: what is the fraction of states of the world (SOWs) in which the criterion is met, or
\[S=\frac{1}{N}\sum_{n=1}^N I_n,\]
where \(I_n\) indicates if the performance criterion is met in SOW \(n\).
Over these ranges, we could calculate a satisfycing score of 41%.
If we shrink the range of \(q\) to be \((2, 3)\), that goes up to 58%.
Which is “right”?
Other satisfycing metrics might measure the “distance” from the baseline case before the system fails.
Regret metrics capture how much we “regret” (or lose in objective value) worse performances across SOWs.
A simple regret metric: what is the average worsening of performance across SOWs?
\[R = \frac{1}{N} \sum_{n=1}^N \frac{\min(P_n - P_\text{base}, 0)}{P_\text{base}},\]
where \(P_\text{base}\) is the performance in the design SOW and \(P_n\) is the performance in SOW \(n\).
\[R_2 = \frac{1}{N} \sum_{n=1}^N \frac{P^\text{base}_n - P^\text{opt}_n}{P^\text{opt}_n}\]
where \(P^\text{opt}_n\) is the performance of the optimal decision in SOW \(n\).
Other options:
Often robustness is used to stress-test an identified decision alternative.
We could also use these metrics during our optimization procedure by:
Just be careful:
These different metrics measure very different things and can rank decisions differently.
Friday: Sensitivity Analysis
After Thanksgiving: No class, will schedule 10-minute meetings with teams during class time the week after Thanksgiving to check on project progress. Attendance is required during your meeting.
Lab 4: Due Friday
Project: Make sure you check what’s due when!