A marketer’s job is to create a demand for a company’s products or services. With competition for consumers being high, marketers use whatever tools they can to attract potential consumers to the company’s ecommerce site, sustain their engagement and convert interest to purchase. Recommender systems are now widely used in ecommerce to suggest items (products or content) that the online user is likely to find relevant and appealing. They do this by learning patterns from data on user behaviour. While this level of personalisation is generally desired by consumers, there is a darker side to personalisation that borders on manipulation. Regulations on responsible use of personal data are established, and there are strong calls for the protection of user well-being. And the anticipated ratification of the CoE AI Treaty will see governments taking stricter measures to protect user autonomy. Marketers increasingly have the difficult task of navigating around ethical AI requirements and business imperatives.
It is expected that customer preference drives choice, and therefore marketers endeavour to understand customer preferences to design campaigns and user experiences around those preferences. However, individual preferences are not as straightforward as they seem, and may even undermine personal well-being. There are two dominant perspectives on what personal well-being means: hedonic and eudaimonic. In short hedonic well-being refers to subjective evaluations of what feels good and what makes life satisfying. It focuses on pleasure, happiness, comfort, and the avoidance of pain or discomfort. It is typically measured through momentary emotions (positive vs. negative affect) and overall life satisfaction. Eudaimonic well-being, by contrast, refers to living well. It emphasises meaning, purpose, personal growth, autonomy, competence, and contributing to something larger than oneself. Rather than asking whether a person feels happy, it asks whether their life feels worthwhile and aligned with their values. For more on this distinction go to https://positivepsychology.com/hedonic-vs-eudaimonic-wellbeing/
Personal preferences can be understood within these two well-being domains. People may prefer opportunities that make them happy in the moment, or opportunities that delay instant gratification for longer term benefit. Preferences can be defined as either revealed, stated or idealised preferences. The revealed preferences approach holds that people’s preferences can be inferred directly from their choices or actions, for example, if someone chooses X over Y, it is assumed that they prefer X. This idea underpins AI mechanisms such as inverse reinforcement learning (IRL) where AI systems infer human preferences by observing behaviour rather than relying on explicit statements. This allows for some consumer autonomy as it enables people’s actions to speak for themselves. Stated preferences are when people explicitly articulate what they like or what they want when prompted. These are used in AI training methods such as reinforcement learning with human feedback (RLHF). Compared to revealed preferences, stated preferences are potentially less influenced by people’s weaknesses and may better reflect people’s long term, stable interests. If short-term or momentary, stated preferences are related to hedonic well-being. Idealised preferences are the preferences a person would have if they were fully informed and free from distorting influences such as misinformation, manipulation, or weakness of will. This aligns them closer to eudaimonic well-being. However, it is sometimes unclear how such preferences will be carried out in practice, which could yield different results. As an example, if someone expresses a preference for eating healthily it is a stated preference, but if they choose unhealthy food items it is a revealed preference. Once they are given all the information on the benefits of healthy food, and they then plan to eat healthier food thereon, it is an idealised preference.
The moral dilemma facing marketers and their marketing technology team is when deciding which preference signals to train recommender systems on, and the digital nudges used to influence online user behaviour. For instance, revealed preferences are not perfect indicators of what people truly want. Choices may be distorted by misinformation, as when decisions are based on false or incomplete information. Or decisions could be based on a weak will or lack of control or compulsive behaviour and addictions. These cases show that behaviour does not always reflect genuine, considered preferences. Recommender systems trained to respond to revealed preferences may be perpetuating behaviour that undermines the well-being of the consumer. A case in point is Temu’s spinning wheel promotion which uses reward psychology that underlies gambling as well as behavioural triggers that steer users towards greater engagement and spending, often in ways that aren’t directly tied to consumer’s genuine shopping goals.
Stated preferences, while reflecting what a person believes they desire, remain imperfect and can be influenced by social desirability, manipulation, misunderstanding or limited knowledge of complex choices. For example, recommender systems that compare a user’s shopping basket choices to that of “people like you bought” can become manipulative when they rely on social proof bias, and exploit cognitive load thereby removing meaningful choice. Consumer well-being is compromised by potential buyer’s remorse, overspending and a reduced sense of control.
There is also the risk that idealised preferences could diverge sharply from a person’s actual preferences, undermining hedonic well-being. Idealised preferences could be influenced by information that comprises unrealistic outcomes, misleading advertising, or socially manipulative choice architectures, resulting in the consumer taking on impossible, inauthentic or unhealthy goals.
Recommender systems should be complemented with other consumer insights approaches that surface broader human values. And marketers should apply ethical AI frameworks when designing ecommerce sites and incorporating digital nudges into customer journeys.
Here are some guidelines to assist marketers on ethical use of recommender systems:
- Preserve user autonomy
Design nudges that empower rather than manipulate, Users should always have the freedom to choose, with clear options to decline or adjust default settings. Avoid deceptive “dark patterns” that pressure users into decisions they wouldn’t otherwise make.
- Promote transparency and informed consent
Clearly communicate when AI agents personalise recommendations or prices. Users should understand how and why certain options are being presented, and what data informs those suggestions.
- Align business goals with consumer well-being
Ensure that nudging strategies create shared value. Encourage choices that are beneficial, sustainable, or genuinely aligned with user interests rather than optimising solely for engagement or sales.
- Ensure accountability in AI-driven nudging
Intelligent agents should be designed with ethical guardrails that prevent them from reinforcing harmful or addictive behaviour patterns. Regular audits of AI logic and reward systems can help ensure alignment with ethical standards.
- Use behavioural insights responsibly
Apply cognitive and behavioural principles such as social proof, anchoring or scarcity with honesty and restraint. The goal is to clarify decision-making, not to exploit psychological biases.
- Provide feedback and opt-out mechanisms
Give users control over their experience. Allow them to adjust recommendation settings, disable nudging features, or report when interactions feel intrusive or manipulative.
Personalisation can be a rewarding experience for consumers, and marketers should seek to respond to consumers’ preferences. However, when manipulative or unethical practices underlie personalised recommendations, the consumer’s rights and well-being is compromised.
Reference:
Hendrycks, D. (2024). Introduction to AI Safety, Ethics, and Society. CRC Press.
