Go for it! If you already have OpenGL experience—doesn't matter what level—you "only" have to add a bit of basic 3D world math and input handling. I found SDL (http://libsdl.org) to be a very good core framework to build upon regarding input handling.
For the 3D world, start with a checkerboard pattern or so. In fact, this is almost exactly what I did to start Angeldust. I once showed off some early game prototypes on my stream, take a look at this one for example:
https://www.youtube.com/watch?v=qMs8YpkSrg0&t=3289
If you keep watching that stream you'll see other in-progress versions so you can see how my product evolved.
For the 3D world, start with a checkerboard pattern or so. In fact, this is almost exactly what I did to start Angeldust. I once showed off some early game prototypes on my stream, take a look at this one for example: https://www.youtube.com/watch?v=qMs8YpkSrg0&t=3289
If you keep watching that stream you'll see other in-progress versions so you can see how my product evolved.