You know that book "JavaScript: The Good Parts"? There needs to be a similar one "AWS: just the good parts". It would probably talk about EC2 (VPS), S3 (cheap bulk storage), SES (emails), and that's about it. When folks get into the elastic-super-beanstalk-container-swarm-v0.3 products, that's when they really kill themselves on the bills.
That said, yes, just using a VPS vendor is the easy way to stick to the good parts.
That said, yes, just using a VPS vendor is the easy way to stick to the good parts.