For the curious: the reason that this property with 24 holds is because 24 = 2^3 * 3. For any prime number p:
p^2 - 1 = (p+1)(p-1)
And p+1 and p-1 must both be multiples of 2 because p is odd. Furthermore, one of p+1 or p-1 is also a multiple of 4 (because they are both multiples of 2 and only 2 apart). So, we can see where the 2^3 factor comes from in the magic number 24. The remaining factor, 3, comes from the fact that p is prime and not a multiple of 3, so either p+1 or p-1 must be a multiple of 3 (otherwise p-1, p, and p+1 would be three consecutive numbers, none of which are divisible by 3, which is impossible).
As a result, for any prime p > 3, (p+1)(p-1) is divisible by 24, so p^2 - 1 is also divisible by 24.
p^2 - 1 = (p+1)(p-1)
And p+1 and p-1 must both be multiples of 2 because p is odd. Furthermore, one of p+1 or p-1 is also a multiple of 4 (because they are both multiples of 2 and only 2 apart). So, we can see where the 2^3 factor comes from in the magic number 24. The remaining factor, 3, comes from the fact that p is prime and not a multiple of 3, so either p+1 or p-1 must be a multiple of 3 (otherwise p-1, p, and p+1 would be three consecutive numbers, none of which are divisible by 3, which is impossible).
As a result, for any prime p > 3, (p+1)(p-1) is divisible by 24, so p^2 - 1 is also divisible by 24.