Understanding the path of data Modern organizations grapple with an avalanche of data from diverse origins. Grasping where this data originates, as well as its consumption patterns, is crucial for assuring its quality and reliability. This is where data lineage emerges as a potent instrument, enabling a clearer overview of data within organizations. Essentially, data […]
Troubleshooting and analysis Regardless of stringent checks, data processes can falter. Here, data lineage shines by assisting teams in pinpointing the origin of errors in their systems, whether in data pipelines, applications, or models. Such pinpoint accuracy drastically trims down the debugging duration, translating to immense time and effort savings. In summary, the systematic mapping […]
The many facets of ML development As we’ve already touched on in our ML chapters, there are several dimensions to consider when thinking about ML at any level. Let’s recap some of the aspects of ML development as they relate to data stewardship: Beyond training – model deployment and monitoring Ensuring model accuracy doesn’t end […]
Transformative architectural principles In order to maximize the benefits of governance, certain foundational principles must be integrated: Zooming in on architectural dimensions Let’s take another look at our five pillars of governance: Summary In our journey through the realms of data, ML, and architectural governance, we’ve underscored the paramount importance of a cohesive strategy for […]
Introduction to the COMPAS dataset case study In the realm of machine learning, where data drives decision-making, the line between algorithmic precision and ethical fairness often blurs. The COMPAS dataset, a collection of criminal offenders screened in Broward County, Florida, during 2013-2014, serves as a poignant reminder of this intricate dance. While, on the surface, […]
Data collection – fetching the textbook data For this study, we’re analyzing a textbook about insects. Let’s fetch and process this data: text = urlopen(‘https://www.gutenberg.org/cache/epub/10834/pg10834.txt’).read().decode()documents = list(filter(lambda x: len(x) > 100, text.split(‘\r\n\r\n’)))print(f’There are {len(documents)} documents/paragraphs’) Here, we’re downloading the text from its source, splitting it into paragraphs, and ensuring we only keep the more content-rich […]
