Comment by jcz_nz

Actually, my experience at least that when dealing with novel problems LLMs fail miserably. Try accessing uncommon API’s - or areas where you’re unsure an API actually exists (REST against Exchange for admin stuff!). Both ChatGPT and Claude produce nice looking solutions dependent on non-existent libraries. Repeatedly.