It's a cool idea to simplify the input as you said, though as @dorkwood pointed out this would lead to bad results with a lot of overfitting, this model needs a lot of variation to only register your facial features as "you", and disentangle that from the surroundings or even from your clothes.
(this is due to how this model learns, maybe a future model will alleviate that)
(this is due to how this model learns, maybe a future model will alleviate that)