It’s really easy to show this even with a known problem. Ask an LLM to play a game of chess, and then give it 1. h3 as a first move. They always screw up immediately, by making an illegal move. This happens because 1. h3 is hardly ever played, so it isn’t part of it’s model. In fact, it’ll usually play a move that ‘normally’ responds to h3, like Bh5 for example

FYI it seems about half of these have been taken down