You seem to be in the camp of believing the hype. See this write up of an apple paper detailing how adding simple statements that should not impact the answer to the question severely disrupts many of the top model’s abilities.

In Bloom’s taxonomy of the 6 stages of higher level thinking I would say they enter the second stage of ‘understanding’ only in a small number of contexts, but we give them so much credit because as a society our supposed intelligence tests for people have always been more like memory tests.