OPTIMIZATION TECHNIQUE (BCS405C)

OPTIMIZATION TECHNIQUE

Course Code BCS405C 
CIE Marks 50
Teaching Hours/Week (L:T:P:S) 2:2:0:0 
SEE Marks 50
Total Hours of Pedagogy 40 
Total Marks 100
Credits 03 
Exam Hours 03
Examination type (SEE) Theory

Module-1: VECTOR CALCULUS

Functions of several variables, Differentiation and partial differentials, gradients of

vector-valued functions, gradients of matrices, useful identities for computing gradients,

linearization and multivariate Taylor series.

(8 hours)

(RBT Levels: L1, L2 and L3)

Module-2: APPLICATIONS OF VECTOR CALCULUS

Backpropagation and automatic differentiation, gradients in a deep network, The

Gradient of Quadratic Cost, Descending the Gradient of Cost, The Gradient of Mean

Squared Error.

(8 hours)

(RBT Levels: L1, L2 and L3)

Module-3: Convex Optimization-1

Local and global optima, convex sets and functions separating hyperplanes, application of

Hessian matrix in optimization, Optimization using gradient descent, Sequential search 3-

point search and Fibonacci search.

(8 hours)

(RBT Levels: L1, L2 and L3)

Module-4: Convex Optimization-2

Unconstrained optimization -Method of steepest ascent/descent, NR method, Gradient

descent, Mini batch gradient descent, Stochastic gradient descent. (8

hours)

(RBT Levels: L1, L2 and L3)

Module-5: Advanced Optimization

Momentum-based gradient descent methods: Adagrad, RMSprop and Adam.

Non-Convex Optimization: Convergence to Critical Points, Saddle-Point methods.

(8 hours)

(RBT Levels: L1, L2 and L3)

Suggested Learning Resources:

Text Books:

1. Mathematics for Machine learning, Marc Peter Deisennroth, A. Aldo Faisal, Cheng

Soon Ong, 2020, Cambridge University Press.

2. S. Bubeck, Convex Optimization: Algorithms and Complexity, Foundations and

Trends in Optimization, 2015.

3. S. Boyd, N. Parikh, and E. Chu,“ Distributed optimization and statistical learning via

the alternating direction method of multipliers”, Foundations and Trends in

Machine Learning, Now Publishers Inc.

Reference Books:

1. Linear Algebra and Optimization for Machine Learning, Charu C. Aggarwal,

Springer, 2020.

2. A. Beck, First-Order Methods in Optimization, MOS-SIAM Series on Optimization,

2017.

3. F. Bach, “Learning with Submodular Functions: A Convex Optimization

Perspective”, Foundations and Trends in Machine Learning, Now Publishers Inc.

Web links and Video Lectures (e-Resources):

• https://ocw.mit.edu/courses/mathematics/18-06sc-linear-algebra-fall2011/index.htm

• https://www.math.ucdavis.edu/~linear/linear.pdf

• https://www.coursera.org/learn/linear-algebra-machine-learning

• https://nptel.ac.in/syllabus/111106051/

• https://github.com/epfml/OptML_course

• https://www.youtube.com/playlist?list=PL4O4bXkI-fAeYrsBqTUYn2xMjJAqlFQzX