| Category | Difficulty |
|---|---|
| HW | 4 |
| Exams | 3 |
| Project | varies |
In Spring 2022, this course was taught by Prof. Qu.
Convex optimization is something that comes up almost everywhere. It's used heavily in control systems for robots or vehicles, for problems like managing a power grid or a stock portfolio, and for signal processing such as reconstructing images. Generally, convex optimization is a set of tools that you can apply to any problem as long as you get it in the right form. Then you can quickly get an answer. This class talks about what those tools are and what those forms are.
Some of the main topics covered here are
- What is convex optimization
- Types of convex problems
- Dual problem formulation
- KKT Conditions
- Gradient Descent
- Stochastic Gradient Descent
- Proximal Gradient Descent
- Projected Gradient Descent (aka a lot of flavors of Gradient Descent)
- Barrier Problems
- Non-convex Optimization
The content for this course is pretty challenging. But if you work to maintain a good intuition of what's going on behind the equations, it can actually make a lot of sense. Particularly at the beginning of the semester, when everything is still gearing up, it pays dividends to keep up with the class. The Canvas quizzes are a helpful way to test yourself and figure out if you're where you should be. For me, this class was pretty unfamiliar. Reviewing the slides between classes and over the weekend helped a lot. If I went 5 days without thinking about the material, the next lecture made much less sense.
There are 4 HWs and 2 midterms. The HWs are challenging, but not particularly long. You have roughly 10 days to 2 weeks for each. They're normally a mix of proofs and finding equations for solving particular convex problems. If you start early and go to OH, they're very manageable. Additionally, there are coding portions for assignments that can be done in python or Matlab. The coding assignments use the CVX library, which is pretty nifty. It's a very different way of solving things, but it can do some impressive things. I'd never done any optimization stuff or worked with CVX before this class, but the pace was well suited for students like me.
The midterms were given as 24hr take home tests. In general, they required around 3hrs each. These are not too hard but require that you've been keeping up with the class. It's similar to the homeworks in format, but easier. It's intended to be completed in 2hrs (but if you're Latexing it could take more like 3). As long as you know all the formulas that have been covered in class and when they hold, you can generally find ways to sew them together for the necessary proofs. For 24hr exams like this, I recommend taking a look at the exam as soon as it becomes available (hopefully it isn't released at 12am. In this class, they were released at class time). This lets you get a good idea of how long it is. You don't want to run out of time if it's longer than you expected.
The course project is very open ended. You have the last month of the course to focus on your project. You can review a paper and reproduce their work or you can find a problem of your own and apply the skills from the class to do something interesting (write scripts, solve the problem, etc...). Projects tend to involve python and/or Matlab code and have varying levels of analytic solutions, depending on the project you pick. Scale-wise, it's roughly average for a final project -- approximately as much work as 2 homeworks, maybe.
Attend classes regularly. Prof Qu also uses Canvas quizzes between classes. These are very useful for checking your own knowledge after each lecture. They can be a good hint that you need to review the lecture more.
Start the HWs early. They aren't too bad, but there is normally at least one question you won't fully get on the first try. It is useful to have a good start so you can get the most out of the office hours.
There are a lot of resources online for learning about Convex Optimization. The author of the textbook, Stephen Boyd, is a professor at Stanford and has his whole Optimization class posted on youtube. These can be really helpful for getting a different perspective if you're confused. That said, the notation and material covered in Boyd's class and in 660 are slightly different, so the course notes are definitely the best support for the HW and exam. For reference, the textbook can be found here: https://web.stanford.edu/~boyd/cvxbook/
Start thinking about a project topic early. If you have a particular research interest, that can be a great place to start. The earlier you can bring the professor an idea, the more refined the idea will be by the time you start and the more the project will flow.
You can check out the course summary here: https://courses.ece.cmu.edu/18660