ChatGPT annotates data 20 times cheaper than humans and has an advantage in 80% of tasks

To put it simply, data annotation is to label the content data on social media, classify it into different themes or concepts, or judge its stance and emotions. These annotated data can be used as training sets or evaluation criteria for NLP models.

Another "human job" has been taken away by AI, and it is closely related to training AI:

Data annotation.

Research from the University of Zurich found that in front of ChatGPT, humans have no advantage in terms of cost or efficiency:

• In terms of cost, the average cost of each label of ChatGPT is less than 0.003 US dollars. , 20 times cheaper than crowdsourcing platforms;
• In terms of efficiency, ChatGPT also "crushes" humans with a 4:1 advantage in tasks such as relevance, stance, and themes.

After the paper was released, some netizens ridiculed that the saying that "generating training data requires manual labor" has become a thing of the past.

Some people even called out, "Is it possible that the restoration and digitization of ancient books will be accelerated?"

Some people watched the excitement and did not take it too seriously. They directly tweeted:

This is directly taking away the jobs of platform workers.

Speaking of which, how did ChatGPT steal the “jobs” of data annotation workers?

ChatGPT has an advantage in 80% of the tasks

First of all, we must first understand the specific content of the data annotation work.

To put it simply, data annotation is to label the content data on social media, classify it into different themes or concepts, or judge its stance and emotions.

These annotated data can be used as a training set or evaluation standard for the NLP model.

In the past, this kind of work was handled manually. For example, MTurk is a crowdsourcing platform that specializes in data annotation.

Within crowdsourcing platforms such as MTurk, there will be a more refined division of labor, such as professionally trained data annotators and crowdsourcing workers.

The former has the advantage of producing high-quality data, but the natural cost is also higher, while the latter, although cheaper, also has quality that fluctuates with the difficulty of the task.

So the research team began to study the potential of large language models (LLM) in this area, and compared the data annotation of ChatGPT (based on GPT-3.5) and MTurk without additional training (zero-shot) performance.

This comparison is based on 2,382 tweet samples previously collected by the research team.

ChatGPT and MTurk respectively mark tweets with five tasks: "relevance, position, theme, policy, and practicality".

There are two evaluation criteria:

• Accuracy: the percentage of correct annotations compared to ChatGPT and MTurk crowdsourcing workers;
• Inter-coders Consistency reliability: Calculated using the consistency between any two of ChatGPT, MTurk crowdsourcing workers and professional data annotators;

The results are also obvious. In terms of accuracy, ChatGPT has Outperforms MTurk crowdsourcing workers on four out of five tasks.

In terms of consistency reliability, ChatGPT surpassed professional data annotators in all tasks.

In terms of cost, as mentioned at the beginning, ChatGPT is 20 times cheaper than manual work on average, not to mention that AI can work 24*7.

However, not all netizens bought into the conclusion reached by the research team. Some people said:

These five tasks are too single. , so is the difficulty. The reliability of such a conclusion based on this alone is questionable.

Some netizens even ridiculed that the research sample was too small:

(actually) only 2,382 tweets were used as a sample.

The "job threat" goes beyond data labeling

Now, it is hard to say whether AI will completely replace a certain type of work, but it will There is no doubt that it affects human work to a certain extent.

Last week, OpenAI released an analysis report stating that 80% of jobs will be affected by ChatGPT to some extent, and 19% of jobs will be seriously affected by ChatGPT.

And occupations with higher salaries will be hit harder.

OpenAI further listed the specific occupations that will be affected, from largest to smallest:

Translation practitioners, text creators (including poets, writers, etc.), Public relations people, mathematicians, tax preparers, blockchain engineers, financial workers, media practitioners...

△图Source: OpenAI

In addition, OpenAI CEO Altman has said on more than one occasion that "AI will replace some of the existing jobs."

Not long ago, the major upgrade of MidjourneyV5 also caused many human painters to say that their jobs were not guaranteed.

emmmmmm, do you think you can still keep your job?

Paper address:​​https://arxiv.org/abs/2303.15056​​​
Reference link:​​​https://twitter.com/arankomatsuzaki/status/1640521970608402435​​

The above is the detailed content of ChatGPT annotates data 20 times cheaper than humans and has an advantage in 80% of tasks. For more information, please follow other related articles on the PHP Chinese website!