Seems like you have your answer then: don't build an info product. Build a product product. With customers.
It will only ever be a full time job if you work on it full time. Since it's your company, you get to choose the hours you work, so you can make your lifestyle look however you want it to look.
So yes, it may in fact be a hell of a lot of work. But fortunately you possess a hell of a lot of time before you die. Divvy it up any way you choose, but please don't give up because it takes effort.
It will only ever be a full time job if you work on it full time. Since it's your company, you get to choose the hours you work, so you can make your lifestyle look however you want it to look.
So yes, it may in fact be a hell of a lot of work. But fortunately you possess a hell of a lot of time before you die. Divvy it up any way you choose, but please don't give up because it takes effort.
The end result is really nice.