Comment by Yajirobe

What happens when it's indistinguishable from a human speaker (in any conceivable test that makes sense)? It's like a philosophical zombie - imagine that you can't distinguish it from a human mind, there's no test you can make to say that it is NOT conscious/intelligent. So at some point, I think, it makes no sense to say that it's not intelligent.