Intrinsically Motivated Learning
SENSEI: Semantic Exploration Guided by Foundation Models
Learning versatile world models
Inspired by children who engage during play in high-level behaviors through imitation, we investigate how to use knowledge embedded in vision-language models as a source of intrinsic motivation in model-based RL. Our method works directly from observed image observations.
Foundation models are a great source of high-level knowledge. We propose SEmaNtically Sensible ExploratIon (SENSEI), a framework to equip model-based RL agents with intrinsic motivation from vision language models for semantically meaningful behavior. We observe more directed exploration towards “reasonable” behavior during free-play that translates into better world-models and improved downstream performance.
Publications
