They don't deny visas "just because they feel like it," nor based on race. If that was the case, no one from Lagos would get a visa.
That said, there is a LOT of fraud out of Lagos. It's in a category of it's own when it comes to US immigration fraud.
You can appeal. It will take several years. In the meantime, it would bode well for your appeal if you moved there to be with your wife. That would show credibility to your marriage, and help to assuage doubts if they suspect it is a green card marriage.