While this tool doesn't work with Windows11 it's pretty useful if you're still on Windows 10 (or earlier). Another tool I wholeheartedly recommend is the 7+ Taskbar Numberer [0]. This should be default functionality built into windows but I'm happy that someone took the time to build it.
[0] https://ramensoftware.com/7-taskbar-numberer