We routinely eat virgin bulls, in fact, it is the best beef we have ever had. Like Doc said, it is slightly leaner and I can't see any difference in the taste. When I was running our feedlot, we had a niche market where we supplied beef to 5 small town meat shops. Every one of them asked me to supply them with meat from yearling bulls if at all possible. Last year, I had a two year old bull separate a front shoulder and it would not heal properly. I was not sure what if the meat would be as good or not, but we have no option but to butcher him. We are still eating this beef and I think it is probably some of the best tasting beef we have ever had. The only problem was that I had to buy another freezer to get all the meat in. We have been selling beef to several city people, and several of them have to me, that they will never buy beef in the store again, after eating the beef we sold them from our bulls. I really don't think you will have any problems if you don't let him breed.
One of my neighbors had to butcher a 5 year old herd bull as he snapped a leg off and could not even get up. This happened in the winter time, so this bull had not bred any females for a few months. He said the meat was excellent. Maybe he got lucky, I am not sure. One butcher I supplied beef to, told me that he loved to buy mature bulls that had been fed for awhile so they were in good condition. He said he could sell roasts from these carcasses faster than anything else he had in his store.