Introduction to Optimization

Lecture 14

September 27, 2023

Review and Questions

Last Class

  • Decision models involve objectives and constraints.
  • Objective:
    • How are we assessing performance? (the metric)
    • Do we want to maximize or minimize?
  • Constraints:
    • What are eligible decisions?
    • Engineering restrictions, performance or regulatory standards, etc.

Wastewater Treatment Model

Last class, we looked at the following model for the treatment of wastewater from two point releases:

\[\begin{alignat}{3} & \min_{E_1, E_2} &\quad 5000E_1^2 + 3000E_2^2 & \\\\ & \text{subject to:} & 1000 E_1 &\geq 500 \\ & & 835E_1 + 1200E_2 &\geq 1459 \\ & & E_1, E_2 &\;\geq 0 \\ & & E_1, E_2 &\;\leq 1 \end{alignat}\]

Solving This Problem

We can solve this graphically (see the JuMP tutorial for example code):

The Solution

So the solution occurs at the intersection of the two constraints, where:

\[E_1 = 0.5, E_2 = 0.85\]

and the cost of this treatment plan is

\[C(0.5, 0.85) = \$ 3417.\]

Questions?

Poll Everywhere QR Code

Text: VSRIKRISH to 22333

URL: https://pollev.com/vsrikrish

See Results

Optimization Models

Components of an Optimization Model

  • Objective Function: The “target” function to be optimized.
  • Decision Variables: Variables which can be changed to affect objective.
  • Constraints: Limits on decision variable values.
  • Feasible Solution: Decision variable values satisfying constraints.
  • Optimal Solution: The “best” feasible solution or solutions (with respect to the objective)

How Do We Solve An Optimization Problem?

Poll Everywhere QR Code

Text: VSRIKRISH to 22333

URL: https://pollev.com/vsrikrish

See Results

Solution Approach 1: Trial and Error

What are some challenges to trial and error?

  • Many possible solutions (infinitely many when a problem is continuous)
  • Feasible region may not be intuitive
  • How do we know when we’ve found an optimal solution?

Solution Approach 2: Generalized Search Algorithms

Function with Multiple Minima

Most search algorithms look for critical points to find candidate optima. Then the “best” of the critical points is the global optimum.

Solution Approach 2: Generalized Search Algorithms

Function with Multiple Minima

Two common approaches:

  • Gradient-based methods
  • Evolutionary algorithms

Generalized Search Algorithms

These methods work pretty well, but can:

  • require a lot of evaluations;
  • may get stuck at local optima;
  • may not converge if step sizes aren’t set correctly

waiting model evaluation meme

Generalized Search Algorithms

These methods work pretty well, but can:

  • require a lot of evaluations;
  • may get stuck at local optima;
  • may not converge if step sizes aren’t set correctly

waiting model evaluation meme

Generalized Search Algorithms

These methods work pretty well, but can:

  • require a lot of evaluations;
  • may get stuck at local optima;
  • may not converge if not tuned well.

waiting model evaluation meme

Constrained Optimization

Unconstrained Optimization

Function with Multiple Minima

Without constraints, our goal is to find extrema of the function.

Can do this using the structure of the objective (gradient) or through stochastic search (Monte Carlo).

Constrained Optimization

Function with Multiple Minima and a Constraint

Now, notice the effect of a constraint!

For a constrained problem, we also have to look along the constraint to see if that creates a solution.

Lagrange Multipliers

We can solve some constrained problems using Lagrange Multipliers!

Recall (maybe) that the Lagrange Multipliers method requires equality constraints. But we can easily create those with “dummy” variables.

Using Equality Constraints

Original Problem

\[ \begin{aligned} & \min &&f(x_1, x_2) \notag \\\\ & \text{subject to:} && x_1 \geq A \notag \\ & && x_2 \leq B \notag \end{aligned} \]

With Dummy Variables

\[ \begin{aligned} & \min &&f(x_1, x_2) \notag \\\\ & \text{subject to:} && x_1 - S_1^2 = A \notag \\ & && x_2 + S_2^2 = B \notag \end{aligned} \]

Using Lagrange Multipliers

Then the Lagrangian function becomes:

\[ H(\mathbf{x}, S_1, S_2, \lambda_1, \lambda_2) = f(\mathbf{x}) - \lambda_1(x_1 - S_1^2 - A) - \lambda_2(x_2 + S_2^2 - B) \]

where \(\lambda_1\), \(\lambda_2\) are penalties for violating the constraints.

The \(\lambda_i\) are the eponymous Lagrange multipliers.

Using Lagrange Multipliers

Next step: locate possible optima where the partial derivatives of the Lagrangian are zero.

\[\frac{\partial H(\cdot)}{\partial \cdot} = 0\]

This is actually many equations, even though our original problem was low-dimensional, and can be slow to solve.

Using Lagrange Multiplers

Despite these challenges, Lagrange Multiplers are at the core of many advanced search algorithms.

Key Takeaways

Key Takeaways

  • We solve decision models through optimization.
  • Decision models have a standard structure:
    • Decision Variables
    • Objective
    • Constraints
  • Goal: find “best” solution with respect to the objective that is feasible.

Key Takeaways

  • Trial and error: not a great approach!
  • Search algorithms: better!
  • Lagrange Multipliers: Finally useful (for constrained optimization)!

Upcoming Schedule

Next Classes

Friday: Solving linear optimization models (linear programming)

Monday: Project proposal peer review.

Wednesday: Guest lecture by Diana Hackett (Mann librarian) on regulations research.

Friday: Lab on linear programming in Julia.