This is an automated archive made by the Lemmit Bot.

The original was posted on /r/machinelearning by /u/jbrinkw on 2024-11-11 04:01:25+00:00.

Is it possible to visualize how an LLM “imagines” a token before and after processing it through the attention layer by feeding the token embeddings into an image model? I understand you can’t copy paste it over, but is there a way to capture the latent transformation caused by the attention layer and apply this transformation to the embedding space of an image model?

For example if i were to enter “poor man,” into an LLM the embedding for “man” would shift toward “beggar” while entering “royal man” it could move closer to “king.” I want to visualize that change. Then you could transfer the embedding for man to an image model and it would create the something like a beggar or a king in this example.

It could make a really cool visualization if you captured the transformation after each attention layer and made a video by interpolating each step.