Preparing the data for modeling Having understood the nuances of the bias and fairness definitions, it’s crucial that we give the same attention to our data preparation process. This involves not just the technical transformations but also a thoughtful consideration of the implications of these transformations on fairness. Feature engineering We’ve already touched on a […]
Final thoughts Remember, while we strive to achieve the best model performance, it’s crucial to constantly revisit the fairness aspect. Addressing fairness isn’t a one-time task but rather an iterative process that involves refining the model, re-evaluating fairness metrics, and ensuring that our model decisions are as impartial as possible. Our ultimate aim is to […]
Data collection – fetching the textbook data For this study, we’re analyzing a textbook about insects. Let’s fetch and process this data: text = urlopen(‘https://www.gutenberg.org/cache/epub/10834/pg10834.txt’).read().decode()documents = list(filter(lambda x: len(x) > 100, text.split(‘\r\n\r\n’)))print(f’There are {len(documents)} documents/paragraphs’) Here, we’re downloading the text from its source, splitting it into paragraphs, and ensuring we only keep the more content-rich […]
