Discussion about this post

User's avatar
Venkatesh Rao's avatar

I think this line of analysis can benefit from an annealing metaphor since that modulates and trades-off between hardness/brittleness and softness. I’ve always thought it’s a good metaphor for how long-term memory cools and settles. Carnival can be understood as an annealing cycle. The first brittle cast can be understood as a model being trained. Language = ore. Training = smelting. Casting = supervised fine-tuning and RLHF. Annealing =inference time use of the model. Not quite coherent metaphor yet but there’s something here. Brian Skinner had a good meditation on it on ribbonfarm back in the day. https://www.ribbonfarm.com/2015/12/10/field-theory-of-swords/

1 more comment...

No posts

Ready for more?