Table of Contents
Inspired by large models
Home Technology peripherals AI Megvii's open source multi-modal large model supports document-level OCR, covering Chinese and English. Does it mark the end of OCR?

Megvii's open source multi-modal large model supports document-level OCR, covering Chinese and English. Does it mark the end of OCR?

Jan 05, 2024 pm 09:23 PM
page Model ocr

Want to convert a document image into Markdown format?

In the past, this task required multiple steps such as text recognition, layout detection and sorting, formula table processing, text cleaning, etc.

This time, it only requires one sentence command,Multi-modal large modelVary directly outputs end-to-end results:

Megviis open source multi-modal large model supports document-level OCR, covering Chinese and English. Does it mark the end of OCR?Picture

Whether it is a large paragraph in Chinese or English Text:

Megviis open source multi-modal large model supports document-level OCR, covering Chinese and English. Does it mark the end of OCR?Picture

Also contains the document picture of the formula

Megviis open source multi-modal large model supports document-level OCR, covering Chinese and English. Does it mark the end of OCR?Picture

Or a screenshot of the mobile page:

Megviis open source multi-modal large model supports document-level OCR, covering Chinese and English. Does it mark the end of OCR?Picture

You can even convert the table in the picture into latexFormat:

Megviis open source multi-modal large model supports document-level OCR, covering Chinese and English. Does it mark the end of OCR?Picture

Of course, as a multi-mode large-scale model, maintaining universal capabilities is essential

Megviis open source multi-modal large model supports document-level OCR, covering Chinese and English. Does it mark the end of OCR?Picture

Vary shows great potential and extremely high upper limit. OCR can no longer require lengthy pipline, directly output end-to-end, and can be customized according to user requirements. The prompt outputs different formats such as latex, word, markdown.

With strong language priors, this architecture can avoid typo-prone words in OCR, such as "leverage" and "dupole". For fuzzy documents, with the help of language priors, it is also expected to achieve stronger OCR effects

The project that attracted the attention of many netizens immediately aroused widespread discussion once it was launched. One of the netizens called out after seeing it, "It's so awesome!"

Megviis open source multi-modal large model supports document-level OCR, covering Chinese and English. Does it mark the end of OCR?Picture

How is this effect achieved?

Inspired by large models

Currently, almost all large multi-modal models use CLIP as the Vision Encoder or visual vocabulary. Indeed, CLIP trained on 400M image-text pairs has strong visual text alignment capabilities and can cover image encoding in most daily tasks.

But for dense and fine-grained perception tasks, such as document-level OCR and Chart understanding, especially in non-English scenarios, CLIP shows obvious coding inefficiency and out-of-vocabularyquestion.

When a large pure NLP model (such as LLaMA) transitions from English to Chinese (a "foreign language" for the large model), because the original vocabulary encoding Chinese is inefficient, the text vocabulary must be expanded to achieve a better performance. Good results.

The research team was inspired by it. It is precisely because of this feature

Now the multi-modal large model based on CLIP visual vocabulary faces the same problem and encounters "foreign language image" ”, such as a page of paper densely packed with text, it is difficult to efficiently tokenize images.

Vary is a solution provided to solve this problem. It can efficiently expand the visual vocabulary without rebuilding the original vocabulary

Megviis open source multi-modal large model supports document-level OCR, covering Chinese and English. Does it mark the end of OCR?Picture

Different from existing methods that directly use ready-made CLIP vocabulary, Vary is divided into two stages:

First, we will use a small only The decoder network generates a powerful new visual vocabulary in an autoregressive manner

Next, in the second stage, the new vocabulary and the CLIP vocabulary are fused to efficiently train the LVLM and give it new The characteristics of Trained on document charts and other data, Vary greatly enhances fine-grained visual perception capabilities.

While maintaining vanilla multi-modal capabilities, it also inspires end-to-end Chinese and English picture, formula screenshots and chart understanding capabilities.

In addition, the research team noticed that the page content that may have originally required thousands of tokens was input through document images, and the information was Vary compressed into 256 image tokens, which also provided information for further page analysis and summary. More room for imagination.

Currently, Vary’s code and model are open source, and a web demo is also provided for everyone to try.

Interested friends can try it~

The above is the detailed content of Megvii's open source multi-modal large model supports document-level OCR, covering Chinese and English. Does it mark the end of OCR?. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
1 months ago By 尊渡假赌尊渡假赌尊渡假赌
Two Point Museum: All Exhibits And Where To Find Them
1 months ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

The world's most powerful open source MoE model is here, with Chinese capabilities comparable to GPT-4, and the price is only nearly one percent of GPT-4-Turbo The world's most powerful open source MoE model is here, with Chinese capabilities comparable to GPT-4, and the price is only nearly one percent of GPT-4-Turbo May 07, 2024 pm 04:13 PM

Imagine an artificial intelligence model that not only has the ability to surpass traditional computing, but also achieves more efficient performance at a lower cost. This is not science fiction, DeepSeek-V2[1], the world’s most powerful open source MoE model is here. DeepSeek-V2 is a powerful mixture of experts (MoE) language model with the characteristics of economical training and efficient inference. It consists of 236B parameters, 21B of which are used to activate each marker. Compared with DeepSeek67B, DeepSeek-V2 has stronger performance, while saving 42.5% of training costs, reducing KV cache by 93.3%, and increasing the maximum generation throughput to 5.76 times. DeepSeek is a company exploring general artificial intelligence

KAN, which replaces MLP, has been extended to convolution by open source projects KAN, which replaces MLP, has been extended to convolution by open source projects Jun 01, 2024 pm 10:03 PM

Earlier this month, researchers from MIT and other institutions proposed a very promising alternative to MLP - KAN. KAN outperforms MLP in terms of accuracy and interpretability. And it can outperform MLP running with a larger number of parameters with a very small number of parameters. For example, the authors stated that they used KAN to reproduce DeepMind's results with a smaller network and a higher degree of automation. Specifically, DeepMind's MLP has about 300,000 parameters, while KAN only has about 200 parameters. KAN has a strong mathematical foundation like MLP. MLP is based on the universal approximation theorem, while KAN is based on the Kolmogorov-Arnold representation theorem. As shown in the figure below, KAN has

Hello, electric Atlas! Boston Dynamics robot comes back to life, 180-degree weird moves scare Musk Hello, electric Atlas! Boston Dynamics robot comes back to life, 180-degree weird moves scare Musk Apr 18, 2024 pm 07:58 PM

Boston Dynamics Atlas officially enters the era of electric robots! Yesterday, the hydraulic Atlas just "tearfully" withdrew from the stage of history. Today, Boston Dynamics announced that the electric Atlas is on the job. It seems that in the field of commercial humanoid robots, Boston Dynamics is determined to compete with Tesla. After the new video was released, it had already been viewed by more than one million people in just ten hours. The old people leave and new roles appear. This is a historical necessity. There is no doubt that this year is the explosive year of humanoid robots. Netizens commented: The advancement of robots has made this year's opening ceremony look like a human, and the degree of freedom is far greater than that of humans. But is this really not a horror movie? At the beginning of the video, Atlas is lying calmly on the ground, seemingly on his back. What follows is jaw-dropping

AI subverts mathematical research! Fields Medal winner and Chinese-American mathematician led 11 top-ranked papers | Liked by Terence Tao AI subverts mathematical research! Fields Medal winner and Chinese-American mathematician led 11 top-ranked papers | Liked by Terence Tao Apr 09, 2024 am 11:52 AM

AI is indeed changing mathematics. Recently, Tao Zhexuan, who has been paying close attention to this issue, forwarded the latest issue of "Bulletin of the American Mathematical Society" (Bulletin of the American Mathematical Society). Focusing on the topic "Will machines change mathematics?", many mathematicians expressed their opinions. The whole process was full of sparks, hardcore and exciting. The author has a strong lineup, including Fields Medal winner Akshay Venkatesh, Chinese mathematician Zheng Lejun, NYU computer scientist Ernest Davis and many other well-known scholars in the industry. The world of AI has changed dramatically. You know, many of these articles were submitted a year ago.

Google is ecstatic: JAX performance surpasses Pytorch and TensorFlow! It may become the fastest choice for GPU inference training Google is ecstatic: JAX performance surpasses Pytorch and TensorFlow! It may become the fastest choice for GPU inference training Apr 01, 2024 pm 07:46 PM

The performance of JAX, promoted by Google, has surpassed that of Pytorch and TensorFlow in recent benchmark tests, ranking first in 7 indicators. And the test was not done on the TPU with the best JAX performance. Although among developers, Pytorch is still more popular than Tensorflow. But in the future, perhaps more large models will be trained and run based on the JAX platform. Models Recently, the Keras team benchmarked three backends (TensorFlow, JAX, PyTorch) with the native PyTorch implementation and Keras2 with TensorFlow. First, they select a set of mainstream

FisheyeDetNet: the first target detection algorithm based on fisheye camera FisheyeDetNet: the first target detection algorithm based on fisheye camera Apr 26, 2024 am 11:37 AM

Target detection is a relatively mature problem in autonomous driving systems, among which pedestrian detection is one of the earliest algorithms to be deployed. Very comprehensive research has been carried out in most papers. However, distance perception using fisheye cameras for surround view is relatively less studied. Due to large radial distortion, standard bounding box representation is difficult to implement in fisheye cameras. To alleviate the above description, we explore extended bounding box, ellipse, and general polygon designs into polar/angular representations and define an instance segmentation mIOU metric to analyze these representations. The proposed model fisheyeDetNet with polygonal shape outperforms other models and simultaneously achieves 49.5% mAP on the Valeo fisheye camera dataset for autonomous driving

Tesla robots work in factories, Musk: The degree of freedom of hands will reach 22 this year! Tesla robots work in factories, Musk: The degree of freedom of hands will reach 22 this year! May 06, 2024 pm 04:13 PM

The latest video of Tesla's robot Optimus is released, and it can already work in the factory. At normal speed, it sorts batteries (Tesla's 4680 batteries) like this: The official also released what it looks like at 20x speed - on a small "workstation", picking and picking and picking: This time it is released One of the highlights of the video is that Optimus completes this work in the factory, completely autonomously, without human intervention throughout the process. And from the perspective of Optimus, it can also pick up and place the crooked battery, focusing on automatic error correction: Regarding Optimus's hand, NVIDIA scientist Jim Fan gave a high evaluation: Optimus's hand is the world's five-fingered robot. One of the most dexterous. Its hands are not only tactile

DualBEV: significantly surpassing BEVFormer and BEVDet4D, open the book! DualBEV: significantly surpassing BEVFormer and BEVDet4D, open the book! Mar 21, 2024 pm 05:21 PM

This paper explores the problem of accurately detecting objects from different viewing angles (such as perspective and bird's-eye view) in autonomous driving, especially how to effectively transform features from perspective (PV) to bird's-eye view (BEV) space. Transformation is implemented via the Visual Transformation (VT) module. Existing methods are broadly divided into two strategies: 2D to 3D and 3D to 2D conversion. 2D-to-3D methods improve dense 2D features by predicting depth probabilities, but the inherent uncertainty of depth predictions, especially in distant regions, may introduce inaccuracies. While 3D to 2D methods usually use 3D queries to sample 2D features and learn the attention weights of the correspondence between 3D and 2D features through a Transformer, which increases the computational and deployment time.

See all articles