To understand the definitions and apply them in practice, a first course in group theory + a basic understanding of vector calculus suffices. To add the adjective "topology", the first parts of a general topology course is enough.
To truly appreciate groups like SO(3), a course in differential geometry and differential topology is useful.
Edit: This is all assuming you have no background in mathematics (or, alternatively, physics) at all. If you do, a targeted text can teach you these concepts in a few pages.
To truly appreciate groups like SO(3), a course in differential geometry and differential topology is useful.
Edit: This is all assuming you have no background in mathematics (or, alternatively, physics) at all. If you do, a targeted text can teach you these concepts in a few pages.