At best, this is a way to sweep unaccountable decision-making under the carpet of "the software said so". Quite likely, though, is that such products will make society a living hell for everyone until these practices are entrenched in the industry and it is too late to roll it all back.
Sounds a lot like redlining to me[0][1].
[0] https://en.wikipedia.org/wiki/Redlining [1] https://www.nytimes.com/2017/08/24/upshot/how-redlinings-rac...