We focus on the reduction of the vibration level of rotors by optimizing the shape of the body. The target is to reduce rotor weight and rotor vibrations leading to higher efficiency and less noise. We consider a finite element discretization of the rotor using a Rayleigh beam model which includes rotary inertia and gyroscopic moments leading to nonself-adjoint systems. We present a general algebraic framework for this case. The mass function is the objective function of the optimization problem and constraints are set on the nonlinear and nonconvex functions of critical speed and unbalanced response. For the numerical solution, algorithms belonging to the class of sequential convex programming are applied for the example of a turbocharger. A remarkable reduction of mass of an initially given prototype could be achieved while significantly reducing the unbalanced response and raising the critical speeds.