How to improve data quality to better meet the needs of AI projects
Translator | Cui Hao
Reviewer | Sun Shujuan
Opening Chapter
In today's society, the development of artificial intelligence has become the focus of global enterprises and governments. However, another problem closely related to artificial intelligence has been ignored: poor data quality.
Artificial intelligence algorithms rely on reliable data to produce optimal results—if the data is biased, incomplete, inadequate, or even inaccurate, the consequences can be devastating.
Artificial intelligence systems that identify patients’ diseases are a good example of poor data quality leading to adverse consequences. When there is insufficient data, these systems can produce erroneous diagnoses and inaccurate predictions, leading to misdiagnosis and delayed treatment. For example, a University of Cambridge study of more than 400 tools used to diagnose Covid-19 found that AI-generated reports were completely unusable due to the use of flawed datasets.
In other words, if the data isn’t good enough, AI initiatives will have devastating real-world consequences.
What does “good enough” data mean?
There has always been a huge debate about what is “good enough” data. Some say good enough data does not exist. Others say that “too good” data can lead to analysis paralysis (Translator: should refer to overfitting) – while HBR points out outright that bad information will cause machine learning tools to fail to work.
At WinPure, good enough data is defined as “complete, accurate, valid, and can be used with confidence in risky business processes, the level of data depends on personal goals and business circumstances.”
Most companies struggle with data quality and governance, even though they won’t admit it. This torture continues to increase the tension of the project and overwhelm them. You can imagine that they are under tremendous pressure to deploy artificial intelligence plans to maintain a competitive advantage. Sadly, issues like dirty data are unlikely to be discussed in the boardroom until it causes the project to fail.
How does bad data affect artificial intelligence systems?
Data quality issues arise at the beginning of the process when algorithms learn based on training data. For example, if an AI algorithm is fed unfiltered social media data, it will extract abuse, racist comments, and misogynistic remarks, as demonstrated by Microsoft's AI bot. More recently, the inability of AI to detect dark-skinned people has also been blamed on problems with training data.
How does this relate to data quality?
Lack of data governance, low awareness of data quality, and siled views of data are the main culprits of poor data quality.
what to do?
When companies realize there is a problem with data quality, they panic about recruiting. By blindly hiring consultants, engineers, and analysts to diagnose and clean data, hoping to solve the problem as quickly as possible. Unfortunately, as the months passed, the problem didn't seem to go away, despite millions of dollars spent. Taking a knee-jerk approach to data quality issues rarely helps.
Real change starts at the grassroots.
If you want your AI/ML project to move in the right direction, take these three key steps.
Recognize and acknowledge data quality issues
First, assess data quality by building a culture of data literacy. Bill Schmarzo is a powerful voice on this, recommending using design thinking to create a culture where everyone understands and contributes to the organization's data goals and challenges.
In today’s business environment, data and data quality are no longer the sole responsibility of IT or data teams. Business users must be aware of issues such as dirty data issues and inconsistent and duplicate data.
So, start by making data quality training a valued organizational effort and empowering teams to identify poor data attributes.
With the checklist below, you can use it to track data quality.
Data Health Checklist
- How to capture, store and manage data?
- How many data sources are connected to your central database, and how well is the data disseminated?
- How well are you managing your data? Have you implemented data governance standards? How much of the data is structured, semi-structured or unstructured?
- How much do you spend manually fixing your data compared to automated data management? How do your teams coordinate with each other when accessing and processing data? Are there frequent internal conflicts between IT and business users?
- What is your data quality status? Is your data timely, complete, accurate, unique and follows standardized rules?
Develop a plan to meet quality metrics
Businesses often make mistakes when it comes to data quality. For example, data analysts are hired to complete routine data cleaning tasks instead of focusing on planning and strategic work. Some businesses use data management tools to cleanse, deduplicate, consolidate and purge data without a plan. Unfortunately, tools and talent cannot solve problems in isolation. Strategies that meet the data quality dimensions are the fundamental solution to the problem.
The strategy must address data collection, labeling, processing, and matching data to AI/ML projects. For example, if an AI recruiting program selects only male candidates for technical positions, then the program’s training data is clearly biased, incomplete (not enough data on female candidates is collected), and inaccurate. Therefore, this data does not serve the true purpose of the AI project.
Requirements for data quality extend beyond the daily tasks of cleaning and repairing data. So, data integrity and governance standards need to be set before starting a project. It saves projects from falling into failure!
Ask the right questions and set accountability
There is no universal standard for “good enough data or level of data quality.” Instead, it all depends on the enterprise’s information management systems, data governance guidelines, knowledge of team and business goals, and many other factors.
But before starting the project, there are a few questions to ask the team:
- What are the sources of our information and what are the methods of data collection?
- What issues can impact the data collection process and threaten positive outcomes?
- What information does the data convey? Does it meet data quality standards (i.e. the information is accurate, completely reliable and constant)?
- Is the designated person aware of the importance of data quality and low quality?
- Are roles and responsibilities defined? For example, who needs to maintain a regular data cleaning schedule? Who is responsible for creating master records?
- Is the data fit for purpose?
Ask the right questions, assign the right roles, implement data quality standards and help your team address challenges before they arise!
Summary
Data quality is more than just fixing typos or errors. It ensures that AI systems are not discriminatory, misleading or inaccurate. Before launching an AI project, it is necessary to address data quality challenges by addressing flaws in the data. Additionally, launch an organization-wide data literacy program to connect each team to the overall goal.
Translator Introduction
Cui Hao, 51CTO community editor and senior architect, has 18 years of software development and architecture experience and 10 years of distributed architecture experience.
Original title:Is Your Data Good Enough for Your Machine Learning/AI Plans?, Author: Farah Kim
The above is the detailed content of How to improve data quality to better meet the needs of AI projects. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
Bytedance Cutting launches SVIP super membership: 499 yuan for continuous annual subscription, providing a variety of AI functions
Jun 28, 2024 am 03:51 AM
This site reported on June 27 that Jianying is a video editing software developed by FaceMeng Technology, a subsidiary of ByteDance. It relies on the Douyin platform and basically produces short video content for users of the platform. It is compatible with iOS, Android, and Windows. , MacOS and other operating systems. Jianying officially announced the upgrade of its membership system and launched a new SVIP, which includes a variety of AI black technologies, such as intelligent translation, intelligent highlighting, intelligent packaging, digital human synthesis, etc. In terms of price, the monthly fee for clipping SVIP is 79 yuan, the annual fee is 599 yuan (note on this site: equivalent to 49.9 yuan per month), the continuous monthly subscription is 59 yuan per month, and the continuous annual subscription is 499 yuan per year (equivalent to 41.6 yuan per month) . In addition, the cut official also stated that in order to improve the user experience, those who have subscribed to the original VIP
Context-augmented AI coding assistant using Rag and Sem-Rag
Jun 10, 2024 am 11:08 AM
Improve developer productivity, efficiency, and accuracy by incorporating retrieval-enhanced generation and semantic memory into AI coding assistants. Translated from EnhancingAICodingAssistantswithContextUsingRAGandSEM-RAG, author JanakiramMSV. While basic AI programming assistants are naturally helpful, they often fail to provide the most relevant and correct code suggestions because they rely on a general understanding of the software language and the most common patterns of writing software. The code generated by these coding assistants is suitable for solving the problems they are responsible for solving, but often does not conform to the coding standards, conventions and styles of the individual teams. This often results in suggestions that need to be modified or refined in order for the code to be accepted into the application
Seven Cool GenAI & LLM Technical Interview Questions
Jun 07, 2024 am 10:06 AM
To learn more about AIGC, please visit: 51CTOAI.x Community https://www.51cto.com/aigc/Translator|Jingyan Reviewer|Chonglou is different from the traditional question bank that can be seen everywhere on the Internet. These questions It requires thinking outside the box. Large Language Models (LLMs) are increasingly important in the fields of data science, generative artificial intelligence (GenAI), and artificial intelligence. These complex algorithms enhance human skills and drive efficiency and innovation in many industries, becoming the key for companies to remain competitive. LLM has a wide range of applications. It can be used in fields such as natural language processing, text generation, speech recognition and recommendation systems. By learning from large amounts of data, LLM is able to generate text
Can fine-tuning really allow LLM to learn new things: introducing new knowledge may make the model produce more hallucinations
Jun 11, 2024 pm 03:57 PM
Large Language Models (LLMs) are trained on huge text databases, where they acquire large amounts of real-world knowledge. This knowledge is embedded into their parameters and can then be used when needed. The knowledge of these models is "reified" at the end of training. At the end of pre-training, the model actually stops learning. Align or fine-tune the model to learn how to leverage this knowledge and respond more naturally to user questions. But sometimes model knowledge is not enough, and although the model can access external content through RAG, it is considered beneficial to adapt the model to new domains through fine-tuning. This fine-tuning is performed using input from human annotators or other LLM creations, where the model encounters additional real-world knowledge and integrates it
To provide a new scientific and complex question answering benchmark and evaluation system for large models, UNSW, Argonne, University of Chicago and other institutions jointly launched the SciQAG framework
Jul 25, 2024 am 06:42 AM
Editor |ScienceAI Question Answering (QA) data set plays a vital role in promoting natural language processing (NLP) research. High-quality QA data sets can not only be used to fine-tune models, but also effectively evaluate the capabilities of large language models (LLM), especially the ability to understand and reason about scientific knowledge. Although there are currently many scientific QA data sets covering medicine, chemistry, biology and other fields, these data sets still have some shortcomings. First, the data form is relatively simple, most of which are multiple-choice questions. They are easy to evaluate, but limit the model's answer selection range and cannot fully test the model's ability to answer scientific questions. In contrast, open-ended Q&A
SOTA performance, Xiamen multi-modal protein-ligand affinity prediction AI method, combines molecular surface information for the first time
Jul 17, 2024 pm 06:37 PM
Editor | KX In the field of drug research and development, accurately and effectively predicting the binding affinity of proteins and ligands is crucial for drug screening and optimization. However, current studies do not take into account the important role of molecular surface information in protein-ligand interactions. Based on this, researchers from Xiamen University proposed a novel multi-modal feature extraction (MFE) framework, which for the first time combines information on protein surface, 3D structure and sequence, and uses a cross-attention mechanism to compare different modalities. feature alignment. Experimental results demonstrate that this method achieves state-of-the-art performance in predicting protein-ligand binding affinities. Furthermore, ablation studies demonstrate the effectiveness and necessity of protein surface information and multimodal feature alignment within this framework. Related research begins with "S
Five schools of machine learning you don't know about
Jun 05, 2024 pm 08:51 PM
Machine learning is an important branch of artificial intelligence that gives computers the ability to learn from data and improve their capabilities without being explicitly programmed. Machine learning has a wide range of applications in various fields, from image recognition and natural language processing to recommendation systems and fraud detection, and it is changing the way we live. There are many different methods and theories in the field of machine learning, among which the five most influential methods are called the "Five Schools of Machine Learning". The five major schools are the symbolic school, the connectionist school, the evolutionary school, the Bayesian school and the analogy school. 1. Symbolism, also known as symbolism, emphasizes the use of symbols for logical reasoning and expression of knowledge. This school of thought believes that learning is a process of reverse deduction, through existing
70B model generates 1,000 tokens in seconds, code rewriting surpasses GPT-4o, from the Cursor team, a code artifact invested by OpenAI
Jun 13, 2024 pm 03:47 PM
70B model, 1000 tokens can be generated in seconds, which translates into nearly 4000 characters! The researchers fine-tuned Llama3 and introduced an acceleration algorithm. Compared with the native version, the speed is 13 times faster! Not only is it fast, its performance on code rewriting tasks even surpasses GPT-4o. This achievement comes from anysphere, the team behind the popular AI programming artifact Cursor, and OpenAI also participated in the investment. You must know that on Groq, a well-known fast inference acceleration framework, the inference speed of 70BLlama3 is only more than 300 tokens per second. With the speed of Cursor, it can be said that it achieves near-instant complete code file editing. Some people call it a good guy, if you put Curs
