There is a need for socially assistive robots (SARs) to pro- vide transparency in their behavior by explaining their rea- soning. Additionally, the reasoning and explanation should represent the user’s preferences and goals. To work towards satisfying this need for interpretable reasoning and represen- tations, we propose the robot uses Analogical Theory of Mind to infer what the user is trying to do and uses the Hint En- gine to find an appropriate assistance based on what the user is trying to do. If the user is unsure or confused, the robot provides the user with an explanation, generated by the Ex- planation Synthesizer. The explanation helps the user under- stand what the robot inferred about the user’s preferences and why the robot decided to provide the assistance it gave. A knowledge-driven approach provides transparency to reason- ing about preferences, assistance, and explanations, thereby facilitating the incorporation of user feedback and allowing the robot to learn and adapt to the user.