It should definitely be added as a variable within the calculation, but the current FSRS predicts how likely you are to access the memory (if it's sufficiently available which is defined by its retrieval strength) and speed of retrieval isn't really a factor in this version. The different grades are more to define how well all parts of the memory is retrieved.
Not to say that how quickly you can access it doesn't play a role in real life.
Not to say that how quickly you can access it doesn't play a role in real life.