The article is a bit rubbish. They're not predicting on the binary "would we give these people a loan or not" but they're predicting manual corrections fo credit scores by bank managers. It's a 15 way classification problem (1 is low score, 15 is high). The data is distributed in a bell-curve like way with the most people in the 6 or 7 bracket.
From the paper:
> As is typical in machine learning we also report the Accuracy p-value computed
from a one-sided test (Kuhn et al., 2008) which compares the prediction accuracy to the "no information rate", which
is the largest class percentage in the data (23.85%).
So fair dice 23.85%, model 95%.
That said I bet a human who had read the banking rules and regulations and recommendations on lending could easily match this performance.
From the paper:
> As is typical in machine learning we also report the Accuracy p-value computed from a one-sided test (Kuhn et al., 2008) which compares the prediction accuracy to the "no information rate", which is the largest class percentage in the data (23.85%).
So fair dice 23.85%, model 95%.
That said I bet a human who had read the banking rules and regulations and recommendations on lending could easily match this performance.