Underlying all cancer therapy protocols are the competing objectives of maximizing tumor control and minimizing normal-tissue complications. As such, we can formulate many aspects of the cancer treatment planning workflow as optimization problems, enabling the development of mathematically rigorous treatment planning methods. In this dissertation, we present three novel optimization approaches to problems in cancer treatment planning: 1) a Markov decision process approach for optimizing multi-modality cancer therapy that balances the trade-off between tumor control and normal-tissue complication, 2) a nonconvex relaxation for the fluence map optimization problem for intensity-modulated radiation therapy that is well adapted to handle nonconvex dose–volume constraints, and 3) a hyperparameter optimization formulation for stereotactic body radiation therapy that has the potential to improve treatment plan quality and reduce the time needed to create a clinically acceptable treatment plan. We demonstrate the feasibility and potential benefit of each approach through numerical examples using synthetic and clinical cancer patient datasets. All project data and code are made openly available on GitHub.