You didn't define small or medium size and number of messages.
From my experience server stream while great for memory and giving the client time to process messages as they come can be slower than unary. One way to solve that is to have a batched message with a repeated field of the underline message you want to send, it's an order faster.
I read sometime ago about about server pushback, maybe you need to override/implement your own StreamObserver to notify the client to slow down as the server isn't ready yet or configure the amount of data the gRPC server can queue there are quite a few configurations (a lot of them obscured) you can make to the server/grpc service.
I read sometime ago about about server pushback, maybe you need to override/implement your own StreamObserver to notify the client to slow down as the server isn't ready yet or configure the amount of data the gRPC server can queue there are quite a few configurations (a lot of them obscured) you can make to the server/grpc service.