mental models of intelligence

Last Updated: 2025-11-23

nostalgebraist model psychology
karpathy on dwarkesh - thinking on working memory, being too good at memorization

karpathy tweet
Claude 4 system prompt (Simon Willison)

some stuff on the other site plopped down here

AI teaches me about myself

We do a lot of crazy things in our meat-ed bodies. Perhaps, they are not crazy after all, but just the natural results of the universe pushing onwards towards some final destiny. However the cause, AI might discourage us from its exceptional ability to solve IMO problems, play games, and write corporate emails and research reports, but along the jagged edges of AI applications, we see there all that we humans uniquely can do. In seeing the intersections of what both of us can do, we also see shadows of ourselves in these machines.

Temperature. At any point in time in the conversation, you have chosen a temperature. There is a more interesting thing to say and a standard conversation pattern. At any point, you may choose to have some fun, or you may choose to conserve energy. If you walk along East Village or run in Central Park, you might be forgiven for thinking you live in a simulation. The conversations recur about someone else’s relationship, about their job, about their next race, about someone else’s job. Fratbros, SWEs, men in finance, girls in East Village, etc are all not born this way. We do not see a convergence to the median (yet, but maybe soon with social media) on a civilizational scale, but we certainly see one on the group scale. It’s called groupthink!

We are LLMs trained on a smaller dataset.

We also have the special ability to constrain our own dataset. You belong to certain groups. Your group-ness induces you to belong to other groups. As you grow older and your “plasticity” firms up, it becomes even unfathomable that you could belong to a different group. Whereas you may have been open-minded to updating your weights, you no longer even consider doing so.

We are multi-dimensional beings who work with more than words. Of course, AI advances towards images and movement, but look at something far away. Do you see, really, the full three-dimensional-ness of your world? How is it possible that someone is so far away? Do you see the shadows, and how the other side wraps around into nothingness, the sharpness of the edges, the brilliance of the colors.

Does the stone Buddha dance for you? Look with the patience of centuries and the precision of the microscopic, Shinzen says. Perhaps Picasso also saw the dynamism and impermanence of the visual field. It contracts with your attention. Details unfold themselves if you ask them nicely, while others are smothered by adjacent richness. There is no ground truth. Your attention is everything.

Yet, we’re rigid, and we don’t realize we are. How many times do we have the flexibility to see something anew? What if that “something” was our existence itself? A lot of self-help is really “what is blocking you from updating your weights?” and I would reckon that we really don’t understand even what our weights are. For example, we think good and true thoughts. We instruct ourselves. Yet we do not change! Foolish as we are. We are wordcels, but our weights are encoded in flesh. The body keeps the score, and our emotions drive our thoughts for the simple reason that we don’t understand our emotions very well. We can make up stories about them, but how would you know when you’re lying to yourself? A lot of problems that you can’t seem to solve are solutions you don’t want to admit to adopting to problems you don’t want to admit to having. The requisite step of change is to change the felt sense, the lived sense, the physical sense, even though changing beliefs tends to be a first, middle, and last step.

AI and us are alike in that we are functions that take in information, process it, and act on it. We are not alike in how we change how we respond to information. We are malleable for a short time, and then our basic structures are formed. Everything becomes fine-tuning.

LLMs and humans alike get better with more feedback. Then what matters is whether we can smartly respond to it. Do we change our weights in small increments? Or do we shift an underlying belief? It’s hard. Shifting beliefs usually requires handling more complexity.

Words are not the world, but my language is my world. LLMs gave me new concepts and have expanded my world.

We take so much for granted.

How would you teach an LLM to decide what is important? It becomes what it reads. What if it was bad? On what basis you decide that it was bad? Is it enough to teach it elementary-school-“blogs are less trustworthy than published papers?” What does cognitive security mean to an LLM?

LLMs also helped me realize that much of what we seek to do are under-specified. We are inspiration-based creatures. We don’t know what we want, but we feel our ways to them. When we write instructions for an LLM, we are so bad at giving it a good supportive context because we don’t know what we want. What does that suggest about yourself? Do you create good supportive contexts for yourself? Or do you leave much of your life up to chance and the hope that the next run will make things better?

Explaining tasks and what you want precisely is very difficult. If you can say, very precisely, exactly what your goal is, then you are most of the way there. It is only in school that “what you want” is clearly defined for you. Past that, you have trouble on both the meta level of “what is worth wanting” and also the project management level of “how do you even get to any of these wants.”

But if you can define that, much is solved.
Contrary to prompting, this need not occur with words. You just need a clear visualization of what you’d like. Tennis enjoyers apparently improve faster when they just try to mimic an instructor instead of taking verbal instructions (how would you teach an LLM? Would you tell them exactly what to do, or just show them how you want it done?).
The LLMs will do the pattern matching, but it is up to you to determine what patterns to match. Creating something new is so much more difficult, but it need not be zero to one. To be inspired means you stand with intellectual lineage. You pair history with modern problems to create something new.

Ina addition to the goal, do you know what an LLM needs? Are you giving it a supportive context?

It is less important to me what exactly reason is, than what reason brings about, and it should be a big revelation that models thinking step by step and thinking more allows them to be “smarter.” We are the same! We think through writing, speaking, reinforcing connections, putting feelings together into something that we can hold on to. That should give you a big “in” about what a good habit to develop is.

We are word machines as well, but we have a life to live. We are agents interacting with other agents. We go through life prompting the outcomes that we want. People are auto-filling based on what they see and what you say and what you do. What is the auto-fill that comes out of your life?

You have beliefs (the words in context), aliefs (the weights), and the context (the project, the task).

That which I learn from AI

Temperature - a more interesting thing to say
Think through writing, retrieval. Think through words, then the map becomes important (both the bijection and the framework)

Explaining tasks and what you want precisely is hard and most of the way there when doing something difficult

Break things down into steps. Then an LLM might be able to do as well. We are just word machines.

model responds diff to diff contrxt kike people, eg cengiz feedback could ring differently

We are mimicry machines (inner game of tennis, trust the data), reward optimizers

we are just LLMs trained on less data and hence have crazier and “crazier” responses

Everyone talks like each other. Oh I met his gf she was cool

Context for answering questions is everything! Creating supportive context.

Inference vs reason (speed)

We are pre-competitiveness?

prompt the responses you want, people are also auto completing

That which AI cannot do

Judgement, determining what is most plausible

What I want to build
Glasses man, hardware

LLM processes info and stores. We have much more info. Can we improve their taste, through time? Accidental human lives, accidental model design? Evolution - ie send them off and simulate them in a market?
A felt sense
How can we compute faster than numbers?
Brain is centralization is natural. More info, processing, and action. Feedback loop.
Foundation and Dune are one. Prophecy and machine.

LLMs can do rote work and things for which there are good answers. Open space is harder to evaluate

on metaphor

What does AI tell us about being human?

Metaphor
This is a worthwhile question to ask, in my opinion, because metaphor is the way we understand the world. If not metaphor, concept. Each lamp produces infinite shades of light tan. Each moment can be split into infinite moments. Each human life consists of uncountable thoughts and emotions.

In order to deal with the infinite complexity of the world, we operate at a layer of abstraction far removed from reality itself. We speak of institutions instead of many groups of people. We talk about a machine instead of all its component parts. We use stereotypes instead of meeting each person where exactly they are.

The vocabulary of the universe is infinite in size. It knows no difference between the discrete and the continuous. In contrast, we are impoverished in our expression. The modern person has also limited themselves to thought as our primary method of processing information (hyperlink), so our language limits what we can think. Anything that limits what we can think then limits what we can believe, what we can act on, what we can live.

So, then, the introduction of new language means an expansion of our world. Recursively, as our world changes, our language expands. Even as we destroy ourselves with modern loneliness or social media, we have enriched our species’ words and the depth of understanding possible.

Alternative Metaphors

Everyone has a system prompt. Know yours before it rules you.

The real questions are at the non-overlaps. Why are LLMs much “worse” at writing than top humans, when they are better at math and coding? The preliminary answer is that math and coding have certain answers that the LLM can optimize for. However, how can it be that humans can be “good” at something that relies so much on judgment, that even they themselves cannot track progress of?
Perhaps this is a measurement error. In other words, people compare LLMs to their favorite authors, but everyone has a different favorite author. If we made as many LLMs as we have writers, then people will certainly have some LLM that they like more than some human author. This also applies to individual pieces of writing. People usually do not know which pieces of theirs will become hit tweets or award-winning books. Given a sufficiently high number of tries, a monkey can write Shakespeare, and we’re just comparing many tries to a few.
Perhaps people are attached to writing for personal reasons that are not just about the writing themselves. The author is not dead. They are alive and well in your “XYZ is my favorite!”
Perhaps humans are actually better at extrapolating based on less clear data. There is good taste and good writing. You develop a taste for it by…reading a lot? And developing judgment? To live life is to collect multi-dimensional data, and perhaps there is something in this multi-dimensional data

You see a math question, and your weights tell you that math questions lead to pain and lead you to flinch away from them. You flinch away from a math question, but your weights tell you that you must do the math question because not completing it correctly leads to pain. You are in conflict, but your weights tell you that conflict means you will not get anything done and that leads to pain.

Theorems

Our data is always contaminated. The loop is an infinite loop upon itself. This is what causes bad mental health persistence and good feedback loop persistence. Fundamentally, it is hard to change our weights because we contaminate our data.
We are LLMs trained on a very small dataset of formal stuff, which is why our “reasoning” tends to sound a lot more like just regurgitating theory instead of thinking about it. How much of a conceptual map do you have about economics?
But we are trained on a very large dataset of being human, which involves feeling and acting and also thinking.
But we are also trained on a very small dataset of language even though we have a rich dataset of being human, which we somehow have no ability to consciously interpret. We can only subconsciously live it. This means no matter the richness in our experience, we talk in the way people around us do.
This means our processing becomes the same, which means our actions and data then become the save. We converge to the median on the group scale. [placeholder for fine-grained logic]
In order to change, you may change any step of this chain.
You can change your actions to get very different data. You may speak at a different temperature, for one. What happens if you leave your training set of language?
You may expand your dataset. Yes, people travel, but also read. Listen to your body. Each of these could be a different essay. [attention, multidimensional]
You may understand your weights and just change them. Listen to your body, again.
LLMs do this in small steps, lest you terribly overcorrect. You ought to too.
There is no objective right and wrong. This is why LLMs cannot decide what is good or bad.
Under-specify the prompt. Give yourself a supportive context. Say exactly what you want / don’t want. This is a poverty of language. But for us it can also just be acting! Even without good data we can prompt ourselves out of things, but because we are actually self-improving, prompting ourselves better can make us better! Depends on chunk size
People can also be prompted. Social interactions, etc. There is something to be said here about what can vs cannot be prompted?
Creating something new is difficult. Raise the temperature. This is where you must reason / invent. Reasoning is in contrast to regurgitatino and memorization, going with the flow. Reason is contrarianism.