Block Relaxation

Abstract

As a gentle introduction to optimization algorithms, we now consider block relaxation. The more descriptive terms block descent and block ascent suggest either minimization or maximization rather than generic optimization. Regardless of what one terms the strategy, in many problems it pays to update only a subset of the parameters at a time. Block relaxation divides the parameters into disjoint blocks and cycles through the blocks, updating only those parameters within the pertinent block at each stage of a cycle. When each block consists of a single parameter, block relaxation is called cyclic coordinate descent or cyclic coordinate ascent. Block relaxation is best suited to unconstrained problems where the domain of the objective function reduces to a Cartesian product of the subdomains associated with the different blocks. Obviously, exact block updates are a huge advantage. Equality constraints usually present insuperable barriers to coordinate descent and ascent because parameters get locked into position. In some problems it is advantageous to consider overlapping blocks.