People self select to be in these situations and they can self select themselves out of it too (by changing employer). There are lots of not so great engineers working in not so great companies. Good engineers tend to have options to improve their work situation and most of them do. Sometimes they are happy to take the money for an easy job as well.
However, I see a lot of issues that have also to do with a few factors that are hard to control:
1) programmers are mostly young. There are older programmers and they don't actually disappear (I'm one of them). However, the number of programmers keeps on doubling every five years meaning that at the ripe age of 46, I've experienced about five doublings meaning there are about 16-32 x more juniors aged 20-25 active than people my age. I work with a lot of people half my age currently.
2) Most people aged 30 would count as "senior" because they are relatively scarce and more experienced than most programmers. I got that label slapped on me around that time. That's nearly 17 years ago. I did not have a lot of experience at the time and I've certainly learned a lot since then. This is a common pattern in our industry. We have a lot of very junior senior engineers rediscovering a lot of things about software engineering every generation because there are very few proper seniors around to learn from. I'm usually the oldest person in a project and that's been the case for almost ten years for me. Yet, I know lots of engineers my age as well and most of them are in a similar situation.
3) Scrum adds roles to teams that I would classify as junior management. Product owner or scrum master would be good examples of job descriptions that end up like that. I've experienced more than a few PMs that were effectively on their first job and scrum masters that had little more than a two day training + a hand shake as management background (been there, done that). Junior management and junior "senior" engineers adds up to very predictable issues. Sometimes it works. Mostly it doesn't. The problem is not the process but a lack of experience on both sides. Hiring more people makes it worse.
4) Proper seniors are expensive but being old doesn't necessarily mean you are very good. I know I have to work hard to keep up with younger people. There's something to say for removing mildly burned out cynical forty somethings from your team. I've had to make calls like that to resolve a few toxic situations. Putting an old person in charge does not necessarily solve your problem either.
5) Software engineers generally lack soft skills needed for management and there's a lot of pressure on people to get promoted to their level of incompetence. Once that happens they are easy to manipulate by senior management. Large organizations with a lot of middle management that isn't so great fall victim to this. It' a classic A's hire A's and B's hire C's. Replace hire with promote and you have basically a template for how great companies almost inevitably turn mediocre. The A's tend to leave when that happens.
6) If you think forty something male programmers are scarce, try finding a forty something female one. They are a small minority in any age group but the golden combination of being forty plus, female, and still doing actual engineering is super scarce. As in: I struggle to name a single person that qualifies as that. Most women I know that age have long stopped being engineers and have turned manager or changed career entirely. Women generally are better at soft skills and make great managers so it's not the worst thing. But there's also sexism that drives them away from engineering prematurely before they've had a chance to become really good at that. I know more than a few women that have hit a glass ceiling where people just naturally assume a lack of technical skills because of gender bias. If you have the option, balance your teams as much as you can and things will get better. Resist the urge to put the girls in charge because they are girls. That's a form of sexism that helps no-one. Instead promote them to being tech lead, senior engineer, principal engineer, etc. They'll make better managers when they are ready for that career move. Better still, they might attract some junior female talent. Women generally know where the good jobs are. I've been in a few places that had good policies on that front and it makes a difference.
However, I see a lot of issues that have also to do with a few factors that are hard to control:
1) programmers are mostly young. There are older programmers and they don't actually disappear (I'm one of them). However, the number of programmers keeps on doubling every five years meaning that at the ripe age of 46, I've experienced about five doublings meaning there are about 16-32 x more juniors aged 20-25 active than people my age. I work with a lot of people half my age currently.
2) Most people aged 30 would count as "senior" because they are relatively scarce and more experienced than most programmers. I got that label slapped on me around that time. That's nearly 17 years ago. I did not have a lot of experience at the time and I've certainly learned a lot since then. This is a common pattern in our industry. We have a lot of very junior senior engineers rediscovering a lot of things about software engineering every generation because there are very few proper seniors around to learn from. I'm usually the oldest person in a project and that's been the case for almost ten years for me. Yet, I know lots of engineers my age as well and most of them are in a similar situation.
3) Scrum adds roles to teams that I would classify as junior management. Product owner or scrum master would be good examples of job descriptions that end up like that. I've experienced more than a few PMs that were effectively on their first job and scrum masters that had little more than a two day training + a hand shake as management background (been there, done that). Junior management and junior "senior" engineers adds up to very predictable issues. Sometimes it works. Mostly it doesn't. The problem is not the process but a lack of experience on both sides. Hiring more people makes it worse.
4) Proper seniors are expensive but being old doesn't necessarily mean you are very good. I know I have to work hard to keep up with younger people. There's something to say for removing mildly burned out cynical forty somethings from your team. I've had to make calls like that to resolve a few toxic situations. Putting an old person in charge does not necessarily solve your problem either.
5) Software engineers generally lack soft skills needed for management and there's a lot of pressure on people to get promoted to their level of incompetence. Once that happens they are easy to manipulate by senior management. Large organizations with a lot of middle management that isn't so great fall victim to this. It' a classic A's hire A's and B's hire C's. Replace hire with promote and you have basically a template for how great companies almost inevitably turn mediocre. The A's tend to leave when that happens.
6) If you think forty something male programmers are scarce, try finding a forty something female one. They are a small minority in any age group but the golden combination of being forty plus, female, and still doing actual engineering is super scarce. As in: I struggle to name a single person that qualifies as that. Most women I know that age have long stopped being engineers and have turned manager or changed career entirely. Women generally are better at soft skills and make great managers so it's not the worst thing. But there's also sexism that drives them away from engineering prematurely before they've had a chance to become really good at that. I know more than a few women that have hit a glass ceiling where people just naturally assume a lack of technical skills because of gender bias. If you have the option, balance your teams as much as you can and things will get better. Resist the urge to put the girls in charge because they are girls. That's a form of sexism that helps no-one. Instead promote them to being tech lead, senior engineer, principal engineer, etc. They'll make better managers when they are ready for that career move. Better still, they might attract some junior female talent. Women generally know where the good jobs are. I've been in a few places that had good policies on that front and it makes a difference.