The traditional method to achieve this is a ssh bastion. You have one tiny server running that has a publicly reachable IP, and can also reach the other servers (usually via the private network). You ssh into the bastion, then ssh from there into your actual server (or use port forwarding to make an ssh connection via the bastion, which some tooling supports out-of-the-box).
Considering the cheapest ec2 instance type is now cheaper than an ipv4 address this is easy to justify if the AWS specific options don't fit your usecase
Considering the cheapest ec2 instance type is now cheaper than an ipv4 address this is easy to justify if the AWS specific options don't fit your usecase