Practical Approaches to Key Information Extraction (Part 2)

Real-Life Key Information Extraction (Part 2): Refining Open-Source LLM Performance

Continuing from Part 1, this guide explores using open-source Large Language Models (LLMs) – Qwen2 2B and LLaMA 3.1 – for invoice information extraction, focusing on overcoming data privacy concerns and hardware limitations (RTX 3060 with 12GB VRAM).

Why Qwen2 2B and LLaMA 3.1?

The choice of these models was driven by resource constraints. Qwen2-VL-2B-Instruct, due to its efficient size, was preferred over larger 7B models. LLaMA 3.1 (8B), accessed via Ollama, was selected for its optimized long-context understanding. Other models, such as Qwen2 in Ollama (lacking image support) and LLaVA (insufficient multilingual capabilities), were deemed unsuitable.

This two-model strategy leverages Qwen2's strength in general key information extraction and LLaMA 3.1's superior long-context handling and JSON output consistency, particularly for multilingual documents. Qwen2 initially extracts raw information, which LLaMA 3.1 then refines and structures into a standardized JSON format. PaddleOCR, as in Part 1, enhances vision capabilities for Qwen2.

A Japanese Invoice Example

A Japanese invoice was used as a test case. The initial OCR process (incorporating language detection and PaddleOCR) yielded the following recognized text:

<code>Recognized Text: 
{'ori_text': '根岸 東急ストア TEL 045-752-6131 領収証 [TOP2C!UbO J3カード」 クレヅッ 卜でのお支払なら 200円で3ボイン卜 お得なカード! 是非こ入会下さい。 2013年09月02日(月) レジNO. 0102 NOO07さ と う 001131 スダフエウ卜チーネ 23 単198 1396 003271 オインイ年 ユウ10 4238 000805 ソマ一ク スモー一クサーモン 1298 003276 タカナン ナマクリーム35 1298 001093 ヌテラ スフレクト 1398 000335 バナサ 138 000112 アボト 2つ 単158 1316 A000191 タマネキ 429 合計 2,111 (内消費税等 100 現金 10001 お預り合計 110 001 お釣り 7 890', 
'ori_language': 'ja', 
'text': 'Negishi Tokyu Store TEL 045-752-6131 Receipt [TOP2C!UbO J3 Card] If you pay with a credit card, you can get 3 points for 200 yen.A great value card!Please join us. Monday, September 2, 2013 Cashier No. 0102 NOO07 Satou 001131 Sudafue Bucine 23 Single 198 1396 003271 Oinyen Yu 10 4238 000805 Soma Iku Smo Iku Salmon 1298 003276 Takanan Nama Cream 35 1 298 001093 Nutella Sprect 1398 000335 Banasa 138 000112 Aboto 2 AA 158 1316 A000191 Eggplant 429 Total 2,111 (including consumption tax, etc. 100 Cash 10001 Total deposited 110 001 Change 7 890', 
'language': 'en',}</code>

This was compared against a ChatGPT baseline, demonstrating ChatGPT's superior performance in this initial test.

Qwen2 2B Results (Standalone)

Using Qwen2 independently revealed its limitations. The JSON output was incomplete and inaccurate in several fields, highlighting its struggles with consistent JSON formatting and long-context processing.

Combined Qwen2 and LLaMA 3.1 Approach

Employing LLaMA 3.1 as a post-processor to refine Qwen2's output yielded improved, but still imperfect, results. While some key fields were accurately extracted, detailed item information remained problematic.

Future Improvements: Fine-tuning Qwen2VL

The next part will detail fine-tuning the Qwen2VL model using a dataset of receipt data, aiming to significantly improve both accuracy and processing speed for this specialized task.

Conclusion

This experiment showcases the potential and limitations of open-source LLMs for real-world key information extraction. While a combined approach offers improvements over using a single model, further refinement, particularly through model fine-tuning, is necessary to achieve optimal performance. The focus on data privacy and efficient resource utilization remains a key advantage of this open-source approach.

The above is the detailed content of Practical Approaches to Key Information Extraction (Part 2). For more information, please follow other related articles on the PHP Chinese website!