Artificial intelligence (AI) algorithms are being increasingly used for automated content generation and decision-making. Recent studies have shown that algorithmic decision-making may be innately susceptible to bias which pertains to the fairness of such decisions[i][ii][iii]. The harm bias causes can be representational or allocative. Representational harm is when a large language model produces stereotypical associations that appear in recommendations, search results, images, text and speech. Allocative harm is when algorithms unfairly distribute or exclude resources, services, opportunities or information between different groups. Numerous frameworks have been developed to detect bias at different stages in the machine learning (ML) pipeline, from pre-processing, in-processing and post-processing2. Drawing from these studies, an integrative 8-point framework was developed that indicates where bias occurs in the machine learning and RAG pipeline, the nature of that bias and the measures used to detect whether that bias has occurred. Remedial measures are also suggested.
The need for algorithmic fairness
Automated decision-making promises to increase productivity and efficiency and has been used in numerous real-world purposes. For example, job applications, loan agreements, product recommendations, and medical diagnoses. Because these decisions emanate from machines, which are considered to be morally neutral and free from subjectivity, outputs are deemed statistically objective, accurate and just. However, cases have arisen where bias and discrimination have been found in machine learning outputs. For instance, Amazon’sML-enabled hiring system was found to be discriminating against female candidates, Google’s ad targeting was sexist as better paid jobs were directed at more males compared to females, Facebook has been under continuous scrutiny by many regulators for instilling biases against classified groups through job ads, Apple’salgorithm to calculate the credit score of customers applying for credit limit increases, was found to be gender-biased, favouring males over females, Uber has been receiving criticism for racial bias as it was using ML algorithms to determine fares based on the suburb status of riders, and Discovery Health has been accused of algorithmic bias against Black healthcare professionals when investigating fraud, waste and abuse.
More scrutiny is being applied to ML outputs, particularly in the finance, healthcare and marketing sectors, and algorithmic fairness is becoming a regulated requirement. The European Union’s General Data Protection Regulation (GDPR), Article 5, Recital 71 states that “personal data must be processed lawfully, fairly and in a transparent manner”. Companies that are bound by the GDPR can face heavy penalties or fines for not complying. Companies that are not bound by such regulations can face reputational damage resulting in loss of customers or clients. Therefore, companies should assess their machine learning pipeline for bias as early in the process as possible.
Causes and measures of bias in machine learning pipelines
A survey by Pessach & Shmueli (2020)1 shows several causes for bias that is occurring in machine learning. Namely, biases already existing in datasets that are used to train models, such as if the data is imbalanced by not representing the target population, in the objectives of the algorithm selected, and by way of proxy attributes where sensitive characteristics can be inferred. Bias enters the machine learning pipeline at different stages. Das et al. (2022)2 posit that in the pre-training phase, existing social biases enter the machine learning process when people decide the intended outcomes of the algorithm, how the data is labelled, and which features to include or exclude. For example, in lending situations choosing features such as income and education over community service and health.
The 8-point framework provides a schematic depiction of the various stages in a machine learning and RAG process and indicates the points at which bias could enter the process.
Fig. 1: 8-point bias framework
Fig 2. Table provides description of the relevant biases and the fairness measures used to detect it.
There are various fairness measures to depict bias at the different stages in the machine learning pipeline. At the start of the process is problem formation, which involves people defining the problem statement, translating it into specific machine learning tasks and establishing the necessary input and outputs. Here cognitive bias can influence such decisions. Korteling et. al (2023)[iv] define cognitive bias as “systematic, universally occurring tendencies, inclinations, or dispositions in human decision making”. In the case of machine learning, cognitive biases such as confirmation bias, in-group out-group bias, or the false consensus effect could influence judgements about what outcomes are preferred[v]. While it is difficult to specify that bias has occurred, and if so, what bias has occurred, a Plus, Minus, Interesting (PMI) analysis can be done to encourage multiple approaches to a problem.
In the classification phase, where data is collected and labelled to denote classes, datasets themselves can be imbalanced. Fairness measures to detect imbalance include Group Imbalance where the number of one group exceeds the number in another group, Class Imbalance where the ratio of positive to negative outcomes in the data differs considerably across groups, and Distributional Imbalance when training data is not evenly distributed across different classes2. A word-embedding association test (WEAT) can detect bias in texts if these are used as the training data. Algorithmic bias occurs during model selection and training and could be the result of the algorithm’s objective function such as loss or cost minimisation. The fairness measures at this phase look at the potential outcome of a model’s decisions. Demographic Parity requires protected and unprotected groups to obtain the same output prediction results with the same probability, Disparate Impactuncovers where members of a protected class based on gender, race, and age are negatively affected. Mitigation requires that the ratio of the positive predictions is similar across groups. Demographic Parity is similar to Disparate Impact but it calculates the difference in positive prediction rates between groups rather than the ratio. Equalised Odds refers to the difference between the false positive rates (FPR) and the difference between the true positive rates (TPR) of the two groups. Equal Opportunity requires TPRs to be similar across groups. It is similar to Equalised Odds but focuses exclusively on TPRs. Individual Fairness requires that similar individuals are treated the same. Counterfactual Fairness determines whether individuals who are otherwise similar but for their protected characteristics are treated in a similar way13.
When the model is deployed it is necessary to continue assessing for bias. This is represented as Predictive Parity by assessing how true outcomes track against predicted ones, and Model Drift which identifies whether a model’s performance has degraded due to changes in the data it is trained on or the relationship between input and output variables. In the case of large language model (LLM) outputs such as content, bias can be detected through a content analysis or by obtaining the views of impacted stakeholders. A Cluster Analysiswill identify underrepresented topics by grouping similar pieces of content together and identifying areas with insufficient information.
Several recent studies have shown that it is not possible to satisfy multiple notions of fairness simultaneously and that there are trade-offs where incompatibility between fairness measures exists. Considerations require that only one of the fairness measures should be chosen1. Importantly, when it comes to including fairness measures into machine learning models, it may compromise accuracy because specific, differentiating data points may have been removed from the training data3 68.
Retrieval-augmented generation (RAG)
RAG shows potential for addressing the limitations of LLMs such as hallucinations, reliance on outdated information and a lack of explainability. By injecting external information into the response, RAG enhances both accuracy and reliability of the generated content. As RAG-based LLMs are increasingly being used in real-world applications with high-stakes consequences, it is vital to ensure their trustworthiness and fairness. In this context, fairness focuses on minimising biases introduced during both the retrieval and generation stages. The external data that is used has a similar susceptibility to bias as training data has. These can be assessed using the measures of Group Imbalance and Equalised Odds. The mode in which the retrieved data is selected and ranked can also significantly affect the fairness of the output. Such as where some information is included or excluded or how it is ranked in importance[vi]8[vii]. The selection and ranking of information impacts knowledge augmentation, which comprises the integration of the external knowledge with the model’s internal knowledge to generate a response. Here, bias can be assessed through Adversarial Prompt Filteringwhich is a prompting technique used to test or exploit vulnerabilities/biases in AI models by crafting inputs specifically designed to confuse, mislead or manipulate the model’s outputs. Counterfactual Filtering is another way of testing the LLM outputs for fairness by comparing a model’s decisions against “what if” scenarios. Actual content that is generated by RAG-based LLMs is assessed for bias as per standalone LLM outputs.
Mechanisms to enhance fairness
The following debiasing mechanisms can be applied to enhance fairness at different stages. During pre-processing, it involves augmenting the training data before it is fed into the machine learning algorithm to increase equal representation, as well as changing labels of the attributes which are used as features or reweighting them. During the in-processing phase, adding fairness constraints into the objective function of the algorithm so that the model optimises for both accuracy as well as for the fairness metric selected. During post-processing, selecting separate thresholds for output scores linked to specific classes so that Demographic Parity is minimised. Counterfactual explanations help to demarcate causal drivers of bias by identifying which protected characteristic such as age, race, and gender leads to certain outcomes by testing different scenarios2. These should generally be excluded from the set of features, but proxies that replace them may still infer these attributes123[viii].
A number of frameworks have been developed to ensure fairness in the RAG process. In the retrieval phase debiasing is approached through LoRA fine-tuning, diverse sampling, and re-ranking. In the generation phase, FairRAG, empirical evaluation, output conditioning, representation adjustment and benchmark evaluation are employed7[ix].
Conclusion
With widespread adoption of AI for automated content generation and decision-making, especially where such outputs could materially affect someone’s well-being, whether psychologically or materially, it is critical to ensure that the potential for bias has been assessed. Bias can creep into a machine learning model at various stages, which can be passed onto the RAG process. The RAG process also presents its own vulnerability to bias at the retrieval and generation phases. There are various assessments for bias that can be applied as well as debiasing mechanisms. The 8-point framework provides a helpful overview of where bias may enter the ML and RAG pipeline and specifies the fairness measure to apply. Other frameworks have been developed to mitigate and remedy bias in RAG processes. By adopting these frameworks, companies can ensure that they are fulfilling ethical requirements as part of their AI governance practices.
[i] Pessach, D., & Shmueli, E. (2020). Algorithmic Fairness. ArXiv, abs/2001.09784. https://arxiv.org/abs/2001.09784
[ii] Das, S., Stanton, R., & Wallace, N. (2022). Alorithmic fairness. Annual Review of Financial Economics, 14, 1-32. https://doi.org/https://doi.org/10.1146/annurev-financial-110921-125930
[iii] Wang, X., Zhang, Y., & Zhu, R. (2022). A brief review on algorithmic fairness. Management System Engineering, 1(7). https://doi.org/https://doi.org/10.1007/s44176-022-00006-z
[iv] Korteling, J. E. H., Paradies, G. L., & Sassen-van Meer, J. P. (2023). Cognitive bias and how to improve sustainable decision making. Frontiers in psychology, 14, 1129835. https://doi.org/10.3389/fpsyg.2023.1129835
[v] Campbell, H., Goldman, S., & Markey, P. M. (2025). Artificial intelligence and human decision making: Exploring similarities in cognitive bias. Computer in Human Behavior: Artificial Humans, 4. https://doi.org/https://doi.org/10.1016/j.chbah.2025.100138
[vi] Ni, B., Liu, Z., Wang, L., Lei, Y., Zhao, Y., Cheng, X., Zeng, Q., Dong, L., Xia, Y., Kenthapadi, K., Rossi, R., Dernoncourt, F., Tanjim, M. M., Ahmed, N., Liu, X., Fan, W., Blasch, E., Wang, Y., Jiang, M., & Derr, T. (2025). Towards trustworthy retrieval augmented generation for large language models: A survey. http://arxiv.org/abs/2502.06872
[vii] Zhou, Y., Liu, Y., Li, X., Jin, J., Qian, H., Liu, Z., Li, C., Dou, Z., Ho, T.-Y., & Yu, P. S. (2024). Trustworthiness in retrieval-augmented generation systems: A survey. https://doi.org/http://arxiv.org/abs/2409.10102
[viii] Ferrara, E. (2023). Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies. Sci,6(1), 3. https://doi.org/10.3390/sci6010003
[ix] Shrestha, R., Zou, Y., Chen, Q., Li, Z., Xie, Y., Deng, S., Vision, I. C. C. o. C., & Pattern, R. (2024). FairRAG: Fair human generation via fair retrieval augmentation 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), http://dx.doi.org/10.1109/CVPR52733.2024.01140
