That's what I do o:
I like doing quests, but if the reward's going to be only low exp/money, I'll skip. If there's fame involved, I might do it. I'll probably go for the quest if it has good material rewards, even if it's out of my way xD I like doing quests for some reason o-o
I normally go back to Eld in the middle of grinding because I don't usually just stay in one place and grind for a long time, LOL. I need a change in scenery, so I'll go back to town for stones and normally turn in quests.
In the end, it's all about whether you want to do them or not. If you don't want to, don't do them xD