When you say that you've observed what has happened over multiple decades, are you observing people within your own social circle and are those people from a diverse and representative background or are they from a narrow demographic? It's true that the relative economic status of white men has declined over the last few decades, but the economic status of women or men from other ethnic groups has grown significantly in that time.
https://www.census.gov/data/tables/time-series/demo/income-p...