Behavior is a product of cognitive processes interacting with the environment. To most accurately predict behavior, a person or AI system has to solve this inversion problem – what cognitive processes led to this behavior? However, AI systems are largely oblivious to the cognitive processes that contribute to behavior. Rather than using past behavior to infer cognition, AI simply uses past behavior and the accompanying environmental conditions to predict future behavior. In other words, AI systems try to solve inversion problems as prediction problems, limiting their generalizability to new environments.
Cognitive models are how we as researchers understand cognition and may endow AI with a similar understanding. The parameters contained within a cognitive model correspond to theorized cognitive processes. While a model may not account for all processes, parameter estimates represent approximations of the processes that we have identified as important to the situation at hand. This may bring a variety of benefits to AI systems. Beyond pure prediction accuracy, model estimates may be more easily interpreted than selected features or raw data, contributing to more explainable AI. Perhaps more importantly, cognitive models provide a unique opportunity to change how AI systems represent people. Consider an AI assistant that learns from only your behavior – it will reinforce your bad habits with no concept of your goals and aspirations. If we let AI understand our goals or preferences through cognitive models it may allow AI to help us move toward the future we desire, rather than a future that reflects the present.