Machine Learning Work Shop- A Proximal-Gradient Homotopy Method for the Sparse Least-Squares Problem

Опубликовано 12 августа 2016, 3:06

Machine Learning Work Shop-Session 5 - Lin Xiao - 'A Proximal-Gradient Homotopy Method for the Sparse Least-Squares Problem' We consider the l1-regularized least-squares problem in the context of sparse recovery or compressed sensing. The standard proximal gradient method (iterative soft-thresholding) has low computational cost per iteration but a rather slow convergence rate. Nevertheless, when the solution is sparse, it often exhibits fast linear convergence in the end. We exploit this local linear convergence using a homotopy continuation strategy, i.e., we minimize the objective for a sequence of decreasing values of the regularization parameter, and use an approximate solution at the end of each stage to warm-start the next stage. Similar strategies have been studied in the literature, but there has been no theoretical analysis of their global iteration complexity. We show that under suitable assumptions for sparse recovery, the proposed homotopy strategy ensures that all iterates along the homotopy solution path are sparse. Therefore the objective function is effectively strongly convex along the path, and geometric convergence at each stage can be established. As a result, the overall iteration complexity of our method is O(log(1/epsilon)) for finding an epsilon-optimal solution.