It's a very specialized program intended to get a computer to do something that computers are generally very, very bad at - write sensible language about a wide variety of topics. Trying to then get that one specialized program to turn around and do things that computers are good at, and expect to do it well, is very silly.

Yes, LLMs are designed to emulate how a human would respond to a prompt by digesting a huge amount of human-generated content. They can do that fairly well except when they can't.