BRIN indexes don't affect the storage layer, so you need to make sure your table is appropriately sorted on disk through other means. CLUSTER can do this as a one-off job (pg_repack if you need to keep the table available for writes), but won't sort new data as it's written.
https://www.postgresql.org/docs/current/sql-cluster.html
https://reorg.github.io/pg_repack/