Introduced is a three-dimensional, physics-based mathematical model capable of efficiently predicting self-excited chatter vibration phenomena in the cold rolling of metal strip and sheet. The described nonlinear chatter model combines the 3D mill structural dynamics behavior with the elastic–plastic rolling process dynamics to predict conditions of instability in a single-stand 4-high mill that can lead to both third-octave and fifth-octave chatter. Formulation of the 3D chatter model is achieved by coupling the dynamic simplified-mixed finite element method with a nonlinear roll-bite process dynamics model to capture self-exciting feedback interactions. In contrast to prior approaches to model chatter in the cold rolling of flat metals, the presented method abandons several simplifying assumptions, including 1D or 2D linear lumped parameter analyses, vertical symmetry of the upper and lower halves of the roll-stack, and continuous contact between the rolls and strip. The model is demonstrated for a single-stand 4-high rolling mill considering the detrimental third-octave self-excited chatter condition. Detailed stability analyses that show time histories of the 3D mill behaviors are presented, respectively, for stable, marginally stable, and unstable rolling speeds, and for changes in the lower housing stiffness to reflect more realistic, asymmetric rolling mill conditions.