State of the art in LLMs + Robotics - 2023 - eviltoast
tldr I write about some of the more interesting works that shaped my understanding of applying LLMs for AI agents and robotic applications. Introduction What is this LLMs as a fad - a caveat Are LLMs actually going to be useful for robotics? Instruct based Benchmarking LLM basics n-shot and reasoning via prompting Chain of Thoughts Self consistency ReAct Tree of Thoughts Automating our automatons Lies, safeguards, and Waluigi Building LLM Agents Finetuning With Tool APIs MRKL Toolformer TALM MM-React Generative Agents: Interactive Simulacra of Human Behavior Socratic Models LLMs and Robotics Code generation and multi agents SayCan Inner Monologues Code As Policies ProgPrompt Statler LM-Nav RT2 Final Thoughts Introduction What is this I’m kicking off a project that is centered around the idea of applying large language models (LLMs) as a context engine; understanding contextual information about the environment and deriving additional steps from human intent to shape its actions.

I was aiming to use LLMs with robotics in an upcoming project, and needed to first verse myself in what is the current must-know techniques in the space. To that end I read a ton of papers and wrote this article to try and suss out the best parts of current state of the art.

I hope this helps people; I’d be thrilled to discuss much of this as well!