Gemini is "reading" your documents in the same sense that google servers are "reading" your documents in order to show them in your browser. Gemini is not a person and won't remember what it has "read".
If you don't want google to handle your documents, then don't put your documents in Google docs.
If you don't want google to handle your documents, then don't put your documents in Google docs.