Who makes your ID card, generates the private key, and stores it there? Do you trust them? Did they retain a copy of the key? Did they implement security effectively (a very difficult task, considering the resources of attackers)? What about securing the manufacturing process?
In the end, you always must trust humans.