Collecting fair data
Your model will be as good as your examples. Collect similar numbers of images for each label, or the model will favour whichever it saw most.
Vary the conditions: different lighting, backgrounds, angles and distances. If every “ripe” photo was taken on a red cloth, the model may learn the cloth.
When the system involves people, variety is a matter of fairness. A model trained mostly on some skin tones, ages or body types works worse for everyone else. Ask who is missing before you train.
All your “ripe” photos were taken on a red cloth. What might the model learn?
Choose the answer you think is right.