This is probably due to the new face recognition stuff which runs in the background in iOS 10 and Sierra. I think I had similar issues for the first couple days as it does an initial run through your library, but I haven't had any issues since. Same with Sierra.
This would line up with other's comments on it magically fixing itself after a few days.
This would line up with other's comments on it magically fixing itself after a few days.