Nós fomos ao restaurante e gostamos muito da comida.

We went to the restaurant and liked the food a lot.

Try this:

We didn't go to X (insert name) restaurant; and even if we had gone there, we probably wouldn't have enjoyed the food anyway.

See, the key word here is the adverb 'probably'. Since you have not been to this hypothetical restaurant, you can only assume what the quality of its food is like, good or bad. To substanciate your assumption, you can say: The cooks overthere are rumored to be very lousy.

Here's the full sentence

We didn't go to X restaurant; and even if we had gone there, we probably wouldn't have enjoyed the food anyway. The cooks over there are rumored to be very lousy.