Optimization Problems and Applications

Acknowledgment

This document is based on the lecture videos by Dr. Emre Sefer, originally delivered as part of the course Machine Learning in Finance at Ozyegin University. The lecture videos can be accessed at the following link: Dr. Emre Sefer’s Lecture Videos.

The content has been compiled and published by FinanceLab.AI. For more information, visit our website: FinanceLab.AI.

Additionally, the material draws on concepts from:

Boyd, Stephen P., and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004. ISBN: 9780521833783.

General Optimization Problem

The general optimization problem is expressed in the standard form as follows:

Minimize:	f₀(x)
Subject to:	f_i(x) ≤ 0, i = 1,…,m,
	h_i(x) = 0, i = 1,…,p,

where:

$x = (x1,...,xn)$ is the vector of optimization variables,
$f0 : ℝn → ℝ$ is the objective function we aim to minimize,
$fi : ℝn → ℝ, i = 1,...,m$ , are the inequality constraint functions that ensure $fi(x) ≤ 0$ ,
$hi : ℝn → ℝ, i = 1,...,p$ , are the equality constraint functions that ensure $hi(x ) = 0$ .

The goal is to find an optimal solution $* x$ that minimizes $f0(x)$ while satisfying all the constraints.

Key Characteristics of the Problem

The functions $f0(x),fi(x),hi(x)$ map an $n$ -dimensional vector $x$ (the input variables) to a scalar value. This mapping is represented as $f0,fi,hi : ℝn → ℝ$ .

For inequality constraints, $fi(x) ≤ 0$ must hold for $i = 1,...,m$ .
For equality constraints, $hi(x) = 0$ must hold for $i = 1,...,p$ .

This general formulation covers various types of optimization problems, and the solution depends on the specific properties of $f0(x)$ , $fi(x )$ , and $hi(x)$ .

Convex vs Non-Convex Optimization

If the objective function $f0(x)$ and all constraint functions are convex, the problem is called a convex optimization problem. Convex optimization guarantees that the solution $* x$ is a global optimal solution.

However, in non-convex optimization problems, the functions may contain multiple local minima. As a result, the solution $* x$ may only be a local optimum, meaning it is not necessarily the best solution globally.

Applications of Optimization

Optimization is a fundamental tool used in various fields, including electrical engineering, financial engineering, machine learning, and biomedical applications. Below are some examples:

Electrical Engineering Applications

Circuit Design: Optimization is used to size devices (e.g., transistors) to minimize power consumption while adhering to manufacturing constraints, timing requirements, and area limitations.
Signal Processing: Problems like filter design are formulated as optimization tasks to achieve the desired signal characteristics.
Communication Systems: Optimization helps in designing transceivers, beamforming, and power control to minimize interference and maximize efficiency.

Financial Engineering Applications

Portfolio Optimization: The objective is to allocate investments across assets to maximize returns and minimize risks (variance). Constraints include budget limits, minimum returns, and maximum investment per asset.
Index Tracking: Instead of buying all constituents of an index (e.g., S&P 500), optimization is used to select a subset of assets that replicate the index’s performance.

Example: Portfolio Optimization

In portfolio optimization, we allocate investments among different assets such as stocks, bonds, or treasury bills. For example:

Let Apple stock account for 40% of the portfolio,
Microsoft stock account for 30%,
A three-month treasury bill account for 30%.

The goal is to minimize the overall risk (variance) of the portfolio while meeting constraints such as minimum return requirements, budget limits, and diversification constraints.

Example: Power Control in Wireless Networks

The signal-to-interference-plus-noise ratio (SINR) at the $i$ -th receiver is given by:

SINR = ∑----piGii----, i j⁄=ipjGji + σ2i

where:

$pi$ is the power used by the $i$ -th transmitter,
$Gij$ is the path gain from transmitter $j$ to receiver $i$ ,
$σ2i$ is the noise power at the $i$ -th receiver.

The problem is to maximize the weakest SINR (minimum SINR across all receivers), subject to power constraints:

piGii maxp im=i1n,...,n∑----p-G--+-σ2, j⁄=i j ji i

subject to:

max 0 ≤ pi ≤ pi , i = 1,...,n.

This problem is typically non-convex, meaning that the solution found may only be a local optimum and is not guaranteed to be the global optimal solution.

Optimization in Machine Learning

Many problems in machine learning are formulated as optimization tasks. Examples include:

Training neural networks involves minimizing a loss function that represents the error between predictions and true values.
Support vector machines optimize a margin to separate data points belonging to different classes.
Clustering algorithms, such as $k$ -means, optimize the allocation of data points to clusters to minimize intra-cluster distances.

Formulating Optimization Problems

An optimization problem has three main components:

Variables: Unknowns to be optimized, e.g., power levels or asset allocations.
Constraints: Represent limitations, such as $fi(x) ≤ 0$ (inequality) and $hi(x ) = 0$ (equality).
Objective: The function $f0(x)$ to be minimized or maximized.

For example, in electronic circuit design, the variables may represent device widths and lengths, the constraints may include manufacturing limits and timing requirements, and the objective may be minimizing power consumption.

Conclusion

Optimization plays a crucial role in various domains. From minimizing risks in financial portfolios to improving power efficiency in circuits and wireless networks, its applications are vast. While convex optimization guarantees global solutions, non-convex problems remain challenging, requiring advanced techniques for satisfactory results.

References

This document draws on material from the following sources:

Dr. Emre Sefer’s lecture videos, which can be accessed at: Dr. Emre Sefer’s Lecture Videos, and published by FinanceLab.AI (FinanceLab.AI).
Boyd, Stephen P., and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004. ISBN: 9780521833783.