Colloidal glasses, bird feathers, and beetle scales can all show structural colors arising from short-ranged spatial correlations between scattering centers. Unlike the structural colors arising from Bragg diffraction in ordered materials like opals, the colors of these photonic glasses are independent of orientation, owing to their disordered, isotropic microstructures. However, there are few examples of photonic glasses with angle-independent red colors in nature, and colloidal glasses with particle sizes chosen to yield structural colors in the red show weak color saturation. Using scattering theory, we show that the absence of angle-independent red color can be explained by the tendency of individual particles to backscatter light more strongly in the blue. We discuss how the backscattering resonances of individual particles arise from cavity-like modes and how they interact with the structural resonances to prevent red. Finally, we use the model to develop design rules for colloidal glasses with red, angle-independent structural colors.