Dont bog down with such thinking. A man's sexual urge or libido depends on his own chromosome and body metabolism. The culture, society, association can make a big impact, but food habit is not one of them. A lot of non veg are lousy on bed, a lot of veg are great. Its upto one's own libido.
You are a veg, do you mean total veg or veg who takes eggs also? No matter which one, foodsupply gives you different components (protein, fat, sugar, carbohydrate etc) as well as energy. When you are a veg, you will be lacking on protein, so taking milk products (milk, cheese, butter) and eggs (if you are allowed) gives the required protein. Where you are well off is the Fat part, it doesnot affect you much. When having sex, as the study says, a 30 minutes sex is equivalent to 15 minutes of jogging and can burn 120 calory of fat. If you are gaining fat from butter, still sex is going to help you to stay fit. Your libido wont get affected in absence of meats or fishes.
Stay cool and learn the art of sex. Dont worry about the food habit on veg or non veg part.