My mom always told me do what you love and the money will follow. Sounds trite but I've spent the last six years going 50/50 between working as a professional skydiver and traveling internationally. Now I am successfully transitioning to web design so I can travel full time.
If it isn't fulfilling then don't do it.