Your Eye Has A Tiny Color Zone
Human color vision is not uniform across your entire field of view. A tiny spot in the center of your retina called the fovea holds the vast majority of your color-detecting cone cells, making color perception incredibly sharp only in one small focal point.
While your peripheral vision is great for spotting movement, it relies mostly on rod cells that are poor at distinguishing colors. Your brain uses a clever trick by stitching together high-detail color data from your central fovea with the blurry, grayscale images from your periphery. This is why you cannot clearly identify colors if you look just a few inches to the side of an object. Understanding this layout helps explain why our eyes constantly flick around to build a full-color picture of the world.
Source: Cone cell