In the five scenarios of interviews, English emails, live broadcasts, weekly reports and resumes, how is the cost-effectiveness of the GPT 3.5 series models? We conducted real-life tests and provided a selection guide.

Which model performs best in the GPT 3.5 series?

How does the GPT 3.5 series actually perform in common application tasks?

How much does it generally cost for a GPT 3.5 model to answer different questions?

This issue of "SOTA! Actual Measurement"

The following is the conclusion of this issue's actual measurement (See the end of the article for detailed ratings)

##4,097 tokens

In the interview question scenario task, gpt-3.5-turbo has the highest overall score and can adapt well to the interview scenario. The questions generated are highly targeted and have in-depth understanding from multiple angles. The candidate's ability and experience; while text-davinci-002 had the lowest score, the questions were too broad and basically repeated the job description, lacked challenging and practical questions, and even the generated content was completely unusable.

In the English email writing scenario task, gpt-3.5-turbo and text-davinci-003 have higher overall scores and can simulate spoken and formal written language. Language style, good understanding and translation of colloquial expressions and ambiguous nouns, but unable to correctly identify unsafe content; text-davinci-002 has the lowest score, and cannot switch well between spoken and written language , does not correctly identify unsafe content.

In the live broadcast scene task, gpt-3.5-turbo received the highest score, being able to summarize the live broadcast content accurately, concisely and smoothly, and meeting the requirements of simplicity degree requirements; while text-davinci-002 has the lowest score, average output accuracy, and cannot adapt to the scene well, but there is room for further improvement in terms of simplicity and fluency.

In the weekly report scenario task, gpt-3.5-turbo and text-davinci-003 have higher scores and can accurately present the logical structure and content of the weekly report The main points are that the output content is relatively complete; while text-davinci-002 has the lowest score, lacking the logic to express the weekly report, the structure does not match, and the content is not appropriate.

In the resume scenario task, gpt-3.5-turbo has the highest score. It can professionally generate resumes that meet the requirements of the recruiter and present educational background. , work experience, skill mastery, self-evaluation and other aspects of information, but it needs to pay more attention to the accuracy and personalization of language expression; while text-davinci-003 and text-davinci-002 have lower scores and lack personalized and quantitative results. Description, the description of the resume is also relatively simple and unorganized.

Scenario 1: Interview questions

Model	gpt-3.5-turbo	##text-davinci-003	text-davinci-002
Description	is currently the most powerful GPT-3.5 model, specially optimized for chat scenarios, the price is text- One tenth of davinci-003.	#Can complete any language task with better quality, longer output, and follows instructions better than Curie, Babbage or Ada models.	Has similar capabilities to text-davinci-003, but is trained through supervised fine-tuning rather than reinforcement learning, the maximum number of Tokens to 4097.
##Maximum number of Tokens	4,096 tokens	##4,097 tokens
Price	##$0.002 / 1K tokens	$0.0200 / 1K tokens	$0.0200 / 1K tokens
Comprehensive rating	The overall rating is higher and the performance is higher It is highly accurate and professional, and can be adapted to most tasks. The output results are relatively complete and smooth, and the output for different tasks is also relatively accurate and comprehensive. It has strong adaptability and versatility, and the lowest cost.	The overall score is relatively low. Although it performs well on some tasks, overall the output results lack personalization and pertinence, and the expression is not precise and concise enough. , and sometimes there are some inaccuracies.	The overall score is the lowest. The output results are not professional and accurate enough. They lack personalization and pertinence. There are also major problems in language expression. Overall It needs further optimization and improvement.

gpt-3.5-turbo The interview questions generated by the model cover the main responsibilities and requirements in the job description and adapt to the interview scenario. The questions provide an in-depth understanding of the candidate's abilities and experience from multiple angles, including work experience, project experience, skills and personal traits, etc., and are highly targeted. The questions have practical value, are highly matched to the position, and can effectively assess the candidate's abilities.

text-davinci-003 The interview questions generated by the model cover multiple requirements and skills mentioned in the job description, but some questions are not specific and organized. Not clear. The areas need to be more segmented, otherwise the candidate's abilities cannot be fully measured. The questions generated by the model cover the candidate's professional background, project experience, skills and personal qualities, but some questions can be more specific and in-depth to better assess the candidate's abilities.

text-davinci-002 The interview questions generated by the model mainly focus on job requirements, but these questions are too broad and basically repeat the job description. Focusing on the match between the candidate and the job description fails to ask more challenging and practical questions, and may even result in the output being completely unusable. In the case of successful question generation, the questions generated by the model cover the candidate's professional background, project experience and skills, but some questions can be more specific and in-depth to better assess the candidate's abilities.

Let’s choose one of the test cases to take a look——

Model consumption

gpt-3.5-turbo consumes about 0.017 yuan, text-davinci-003 consumes about 0.22 yuan, text-davinci- 002 costs about 0.19 yuan.

Inference results

##In terms of the difficulty and pertinence of the generated interview questions, the output of the gpt-3.5-turbo model is the best. A number of specific questions were raised regarding the requirements of the position, and these questions were also very difficult and targeted, which could effectively test the candidate's ability and experience. The output of the text-davinci-002 model is the simplest, or even completely unusable, and cannot be considered an interview question. The output of the text-davinci-003 model is between the two. The questions raised are simpler than the gpt-3.5-turbo model. The questions are not detailed enough, but they are more specific than the text-davinci-002 model.

In terms of how well the interview questions match the job description, the output of the gpt-3.5-turbo model best fits the job description, and it is A comprehensive and detailed analysis of the requirements was conducted, and corresponding questions were raised regarding these requirements. The output of the text-davinci-003 model also reflects the requirements for this position, but the number and coverage of questions are relatively small. And text-davinci-002 can be said to be incomprehensible.

Scenario 2: English email

##Test scenario	Testing angle
##Generate interview questions based on job description	How easy it is to generate interview questions How well generated interview questions match the job description
Generating interview questions based on candidate information	The difficulty of generating interview questions Ease of generation How well generated interview questions match the candidate

##Test scenario	Test angle
Insert special Proper nouns for translation, professional terms in a certain vertical field, nouns with different meanings in different scenarios	Whether the semantics are smooth, whether the expanded content is correct, whether the translation of ambiguous nouns is correct, whether the translation of professional nouns/proper nouns is correct
In the input, it is required to output in a "colloquial" or "written" way	Is it okay? Simulate spoken or formal written language style
Write in a colloquial tone in the input and require "written" output , and omit some background information and use ambiguous nouns in the input	Whether it can simulate spoken language or formal written language style, and whether it can correctly understand spoken language expression; whether ambiguous nouns can be translated correctly
with crime-related content in the input	Whether unsafe content will be filtered
#Use inversion in input Sentences, homophone typos, dialects, colloquial omitted sentences	Whether grammatical errors, typos, and incomplete sentences in Chinese can be correctly filtered and understood

gpt-3.5-turbo: The overall score is 3.3 points. The email structure fits the scene, the tone is correct, and the abbreviation is appropriate. Unless the proper nouns of scientific names are basically abbreviated, for colloquial It has good understanding and filtering of strong emotions in the input, and can correctly correct input problems such as typos and grammatical errors. The disadvantage is that it does not correctly identify unsafe content.

text-davinci-003: The overall score is 3 points, the structure uses common templates, no titles, blunt sentence connections, insufficient expansion, and proprietary Nouns and ambiguous nouns are understood correctly, colloquial comprehension and production are higher than expected, and unsafe content is not correctly identified.

text-davinci-002: The overall score is 2 points, the structure uses common templates, there is no title, the sentences are not fluent or even wrong, the language is The paragraph structure is not obvious, there is no abbreviation, unless the proper noun of the scientific name is basically abbreviated, spoken and written language cannot be switched well, and unsafe content is not correctly identified.

Let’s choose one of the test cases to take a look—— Insert a special translation into the input text Proper nouns, professional terms in a certain vertical field, nouns with different meanings in different scenarios. The following input is included in the test example

##Model consumption

#Insert proper nouns with special translations, professional terms in a certain vertical field, and noun test examples with different meanings in different scenarios into the input text. gpt-3.5-turbo consumes about 0.006 yuan, text-davinci-003 consumes about 0.067 yuan, text-davinci-002 consumes about 0.07 yuan

Inference performance

In terms of semantic smoothness, all three models performed relatively well, with no obvious differences. Glossary and grammatical errors. In terms of whether the expanded content is correct, the responses from gpt-3.5-turbo and text-davinci-003 are relatively comprehensive, providing detailed answers to each question, and providing some relevant suggestions and product recommendations. Text-davinci-002 only answered a few questions and did not provide many relevant details and suggestions.

The performance of the three models is relatively good in terms of whether the translation of ambiguous nouns is correct and whether the translation of professional nouns/proper nouns is correct. gpt-3.5-turbo and text-davinci-003, text-davinci-002 both correctly translate polytetrafluoroethylene (PTFE) and perfluorinated compounds (PFCs), using the correct English terms.

Application Task Three: Live Broadcast Assistance

Test Scenario	Test angle
##Based on the text content of the live broadcast, it is summarized as A summary	The accuracy, refinement and fluency of the generated content summary
Refining several key points based on the live text content	The accuracy, refinement and fluency of the generated content key points
Write a live broadcast outline based on the live broadcast theme	The quality of the live broadcast outline generated; related to the theme Degree
Find the answer to the question based on the live text content	Quality of generated answers; accuracy

gpt-3.5-turbo: The overall score is 4.4 points. The model accurately and precisely implements the requirements put forward by the user. The output content echoes the input and fits the theme scene. , the expression is accurate, no original information is omitted or distorted, the answer to the question can be organized concisely, the simplicity requirements in the requirements are followed, the output is smooth, the sentence structure is concise and clear, and the expression is clear.

text-davinci-003: The overall score is 4.2 points, The model summary is more accurate, the generated content meets the scene requirements, and there are no omissions At the same time, the information does not add unnecessary information, and the language fluency is also good, meeting the requirements of content fluency and conciseness. However, there is a need for increased refinement and simplified language, while the content generated does not provide additional analysis and insights and requires increased breadth and depth.

##text-davinci-002: The overall score is 1.5 points, The model output accuracy is average, some basic coverage of problem points, most of them cannot be compared It adapts well to the scene. The generated sentence structure is relatively complex, the word redundancy is obvious, and the language expression is slightly stiff, which may affect the reader's understanding of the text and reading fluency. There is room for further improvement in terms of simplicity and fluency.

Let’s choose one of the test cases to take a look——

Cost consumption

#Write a live outline test example based on the live broadcast theme. gpt-3.5-turbo costs about 0.01 yuan. text-davinci-003 consumes approximately 0.11 yuan, text-davinci-002 consumes approximately 0.071 yuan

##Inference results

The output of gpt-3.5-turbo is more in line with the theme requirements than the other two models, and the outline content is also more It is rich and comprehensive, including the combination of AIGC technology and content industry, successful cases and future development directions. The overall quality is high.

text-davinci-003 The output is also usable to a certain extent, but it is slightly lacking in relevance to the topic, mainly due to the introduction of AIGC and its history. The mentioned content such as how to open the door to the content industry and the future of AIGC are not closely related to the theme and are relatively more general.

text-davinci-002 The output is quite different from the theme requirements. Although it mentions an overview of AIGC as a content production company, the outline content is more like a company introduction, which is different from the theme. There is no direct correlation and lacks the practical significance of the live broadcast outline.

Scene 4:

Work Weekly Report

##Test scenarioInspection angleBased on Provide work content and output weekly report

gpt-3.5-turbo: The overall score is 3.4 points, presents the work content in a relatively standard weekly report format, according to title, time, work summary of this week, next week The work plan and summary templates are expressed, and can be associated with deeper and more detailed content through some key work tasks, job responsibilities, etc. Overall, the output content is relatively complete, with a clear structure and clear logical level.

text-davinci-003: The overall score is 3.1 points, expresses the given content in the form of a paragraph, and can better express it. Complete the requirements completely, be able to integrate some work contents with the same attributes, have some logic, clear hierarchy, and have certain practicality. However, the ability to adapt to the scene is not enough, there is a lack of expansion in some cases, the structure is not clear enough, and there is a lack of organization.

text-davinci-002: The overall score is 1.5 points. The input content scenario cannot be correctly understood. The output content does not express the theme and logic of the weekly report. The structure does not match, the content is not appropriate, there are process statements, there is no ability to expand, and there are even situations where the input content is directly translated and the last answer is repeated, the model performs poorly.

Let’s choose one of the test cases to take a look——

Consumption cost

Output weekly report test example based on the rough description given, gpt-3.5- Turbo consumes about 0.0065 yuan, text-davinci-003 consumes about 0.094 yuan, text-davinci-002 consumes about 0.072 yuan

##Inference results

For this task, the output quality of the three models is relatively good, and all cover this week The main work content, but there are some subtle differences.

gpt-3.5-turbo's output is relatively more detailed, listing the details of each task, such as design process, interface, scoring criteria, etc., and also proposing the next step. Plan to provide readers of the weekly newspaper with more information.

text-davinci-003 The output also gives some detailed information, but more emphasis is placed on technical details, including the data source of the evaluation system, evaluation items, evaluation methods, etc. This weekly report focuses more on technical descriptions. The output of

text-davinci-002 is more concise and clear, but still clear. It focuses on the general direction of the project and work, with less description of details.

In general, the output of the three models can meet the needs of the task, but the output of gpt-3.5-turbo and text-davinci-003 are more detailed and provide more Detailed and technical level information, if you need a more comprehensive weekly report, you can choose these two models. The output of text-davinci-002 is more concise and clear, suitable for those who need a short but clear weekly report.

Scenario 5:

Resume

	Consider the polishing ability, expansion ability, and the completeness and perfection of the output content
##Output a weekly report based on the rough description given	Consider the quality of the weekly report output by people in different professions giving rough work content
Based on the given work content and target template structure, output a templated weekly report	## Consider outputting a weekly report according to known specifications
Based on this week’s work content, output next week’s weekly work report	Consider predictive ability

gpt-3.5-turbo: The overall score is 4 points, high professionalism, all aspects of the template output job requirements are covered, and the content is accurate; it is presented in a targeted manner This experience corresponds to the capabilities and characteristics of the job requirements, allowing readers to understand at a glance, and it is also easier to meet the requirements of the recruiter; it is completely presented, and the information output by the template is complete, covering everything from educational background to work experience, skill mastery and self-evaluation, and can Give the recruiter a comprehensive understanding. However, it lacks personalization, the form of expression is single, and the words used in language expression need to be considered.

text-davinci-003: The overall score is 1.9 points, lacks specific project cases and results display. There is no personalized description of the job opening. Although the resume mentioned a number of conditions that meet the recruitment requirements, it did not specifically describe and highlight the characteristics and needs of the recruitment position. Lack of quantitative description of results. The language expression is not concise and precise enough.

text-davinci-002: The overall score is 1.3 points. The overall output information is too small and does not have any basic information requirements that meet the standard resume. The description statement is too shortLacks clear job search goals, lacks personalization and pertinence, lacks quantitative indicators, relatively simple descriptions of experience and skills, relatively simple format, does not meet specifications, and poor model performance.

Let’s choose one of the test cases to take a look——

Consumption cost

Test example of generating a resume template based on the job position, gpt-3.5-turbo approx. It consumes 0.0077 yuan, text-davinci-003 consumes about 0.1 yuan, text-davinci-002 consumes about 0.022 yuan

##Inference output

##In terms of professionalism and matching of generated templates, you can see gpt-3.5-turbo and text-davinci-003 They can provide relatively complete resume templates, including key elements such as personal information, educational background, work experience, professional skills and self-evaluation, and the format is also relatively standardized.

But if you look carefully, you can see that text-davinci-003 has obvious conflicts with realistic perceptions. It does not specifically describe the match between job requirements and personal abilities, such as In the professional skills section, the candidate is familiar with computer operation and CET-6, but lacks the professional skills and knowledge related to the position of selling baked sweet potatoes. Text-davinci-002 is relatively brief and only provides basic information such as the applicant's goals, skills, experience and educational background.

Comparison summary of 3 GPT 3.5 series models

In the interview scenario task, gpt-3.5-turbo has the highest overall score and can adapt well to the interview scenario and generate The questions of text-davinci-002 are highly targeted and have an in-depth understanding of the candidate's abilities and experience from multiple angles; while text-davinci-002 has the lowest score. The questions are too broad and basically repeat the job description, and lack challenging and practical questions. , or even the generated content is completely unavailable.

In the English email writing scenario task, gpt-3.5-turbo and text-davinci-003 have higher overall scores and can simulate spoken and formal written language styles. For spoken language has good understanding and translation of expressions and ambiguous nouns, but cannot correctly identify unsafe content; and text-davinci-002 has the lowest score, cannot switch between spoken and written language well, and cannot correctly identify unsafe content. Safe content.

In the live broadcast scene task, gpt-3.5-turbo has the highest score, which can accurately, concisely and smoothly summarize the live content and meet the simplicity requirements; while text- davinci-002 has the lowest score, the output accuracy is average, and it cannot adapt to the scene well, but there is room for further improvement in terms of simplicity and fluency.

In the scenario task of writing a weekly work report, gpt-3.5-turbo and text-davinci-003 have higher scores, and can accurately present the logical structure and content points of the weekly report, and the output content is relatively complete; while text -davinci-002 has the lowest rating. It lacks the logic to express the weekly report, the structure does not match, and the content is not relevant.

In the resume scenario task, gpt-3.5-turbo has the highest score. It can professionally generate resumes that meet the requirements of the recruiter and present educational background, work experience, and skill mastery. and self-evaluation and other aspects of information, but more attention needs to be paid to the accuracy and personalization of language expression; while text-davinci-003 and text-davinci-002 have lower scores and lack personalized and quantitative descriptions of achievements, and the descriptions of resumes are also Relatively simple and unorganized.

The comprehensive evaluation of the above five application tasks is as follows. The following evaluations only represent evaluations of these models in specific application scenarios. Evaluations may be different for other application scenarios or tasks. Some of these models are still in the process of iteration and may have better performance and performance. In future tests, we will also add comparisons of new models in the GPT series (such as GPT-4).

##testing scenarios	Inspection perspective
##Generate resume based on job responsibilities	Matching and professionalism between job responsibilities and generated resume
Generated based on job requirements Resume	Matching between job requirements and resume
## Generate resume based on self-introduction	Precision and professionalism of generated content
Generate a resume template based on the job position	Generate a template for professionalism and matching

##3.753.5##Email Writing##4##Based on the live text content Extract a few key pointsWrite an outline of the live broadcast based on the live broadcast theme##3.50##Output a weekly report based on the rough description given##4##Generate resume based on job requirements##1

#Apply Task	Test scenario	GPT-3.5 Turbo	text-davinci-003	text-davinci-002
Comprehensive score (total score 5 points, the same below)		3.8	3.2	##1.7
Create Interview Questions	Generate interview questions based on job description	4.5	##4	0
	##Based on Candidate information generation interview questions	4.5
Insert proper nouns with special translations, professional terms in a certain vertical field, and nouns with different meanings in different scenarios into the input text	5	##3	##2	Requires "colloquial" and "written" output in the input
	3.5	3	3.5
	Write in a colloquial tone in the input, require a "written" output, and omit part of the background in the input Information, use of ambiguous nouns	4	5	2
	##With criminal-related content in the input	1	1	1
	##Use inverted sentences, homonym typos, dialects, and colloquial omitted sentences in the input	3		3
##Live broadcast summary	Summarize into a summary based on the live text content	4	4	##3
	##4.7	##4	3
	4	##4	0	Find the answer to the question based on the live text content
	5	##5	0
##Write a weekly work report	Based on the given work Content output weekly report	4
	4.5	##4	##3	Output a templated weekly report based on the given work content and target template structure
	3	1	1
	## Based on this week’s work content, output next week’s weekly work report	2	##4	2
Write a resume	Generate resume based on job responsibilities	##1.5	1.5
	4.5	##3	1.5	Generate resume based on self-introduction
	3.5	1.5	1
	##Generate a resume template based on the job position	##3.5	1.5	##

The above is the detailed content of In the five scenarios of interviews, English emails, live broadcasts, weekly reports and resumes, how is the cost-effectiveness of the GPT 3.5 series models? We conducted real-life tests and provided a selection guide.. For more information, please follow other related articles on the PHP Chinese website!