This talk will discuss the ability of Large Language Models (LLMs) to figure out the limits of language. Building on the metaphor that construes models as bags of words that approximate human linguistic cognition in increasingly plausible ways, the results of two experiments that tap into grammatically (im)possible language and lexical decoding respectively will be presented. The results suggest that while the tested models are convincingly approximating human baselines, there are qualitative and quantitative differences that reliably distinguish humans from models. From a theoretical perspective, it seems that LLMs are sensitive to a distinction between semantically impossible concepts vs. impossible structures, granting support to theoretical models that posit that there are different sets of rules, and thus of violations, that come from syntax vs. semantics. A systematic investigation of the alignment of Large Language Models to human performance is likely to provide cross-linguistically robust results that point out to those aspects of human language that are still not within the reach of the current generation of Large Language Models.