Sometimes it's the color, and sometimes it's the size of the bait/lure that the fish are keyed in on. If you can determine which is the more important factor early on, it can make a fair trip into a great trip. Here is a quick example:
During an Everglades camping trip a few years back, my friend Steve and I wore out the snook on size 8 X-Raps in any color we threw (Truth be told the snook wore out our lures!). The (at the time) new small x-rap was the perfect match to the small baits the snook were keyed in on, but the color made almost no difference. Throwing a larger X-Rap in the same color produced no bites. The snook wanted small baits, and didn't care about the color.