If you have a worldwide CDN yourself, or you only care about local users, or your fonts are small (Latin subset), yes.
If you're using CJK fonts and you don't already have a CDN for your stuff, no. Google Fonts' CDN is going to beat your own server from any other continent.
If you're using CJK fonts and you don't already have a CDN for your stuff, no. Google Fonts' CDN is going to beat your own server from any other continent.