There are other toxic substance like phosphorus in "lithium" batteries. Some "lithium" battery chemistries, though reportedly not LFP, contain cobalt, a very toxic heavy metal. Besides, there isn't a clear definition of what is a heavy metal. Even iron and copper, used in LFP, are considered heavy metals by some.
BTW, lithium itself is highly toxic too.