I think it depends on whether you use it for operations or for data analysis. Speed is only one concern and it may not always be the most relevant concern.
A statistician/data scientist wrangling data and making plots would not have cared whether loading a CSV file takes one second or one microsecond, because they may only do it a handful of times for a project.
A data engineer has different requirements and expectations. They may need to implement an operational component that process CSV files repeatedly for billions of time a day.
If your use case is the latter, then pandas is probably not for you.
A statistician/data scientist wrangling data and making plots would not have cared whether loading a CSV file takes one second or one microsecond, because they may only do it a handful of times for a project.
A data engineer has different requirements and expectations. They may need to implement an operational component that process CSV files repeatedly for billions of time a day.
If your use case is the latter, then pandas is probably not for you.