Answer: The problems would not have been caused by cross-pollination because that only affects plants grown from the seeds of the cross pollinated crop. In all probability the problems were caused by either the varieties you grew and/or the cultural conditions you (and nature!) provided. If you can grow tomatoes, you should be able to grow peppers, too. Peppers need warm soil and warm sunny weather and adequate moisture to bear well, but there are lots of things that can go wrong including the following:
Common reasons for poor pepper production are setting out transplants when the soil is still too cool, using transplants which have not been adequately hardened off, or setting out poor quality transplants including those which have been held too long and are stunted as a result.
After that, temperatures may be either too cool (under 55 degrees) or too hot (over 75 at night, over 90 during the day). Next, the plants might be over-supplied with nitrogen or watering may be inadequate. Peppers need a good soaking about once a week and like a moist but well drained (not soggy) soil; they also appreciate a good layer of organic mulch.
Finally, they need to be planted in full sun for best production.
You might also find that the "heat" increases as the peppers ripen. While peppers may be picked over a long period and at various stages of ripeness, many types take a very long time to mature fully -- it depends on the variety, so that is something to make a note of when you plant. Good luck with your next crop!
Q&A Library Searching Tips