Building a Meta Tags Scraping API in Under Lines of Code

Have you ever wondered how messaging apps like Whatsapp or Telegram let you see a preview of a link that you send?

Whatsapp and Telegram url previews

In this post, we'll be building a scraping API with Deno that accepts a URL and retrieves the meta tags for it, so we can get fields like the title, description, image and more from almost any website.

For example:

curl https://metatags.deno.dev/api/meta?url=https://dev.to

will give this result

{
  "last-updated": "2024-10-15 15:10:02 UTC",
  "user-signed-in": "false",
  "head-cached-at": "1719685934",
  "environment": "production",
  "description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "keywords": "software development, engineering, rails, javascript, ruby",
  "og:type": "website",
  "og:url": "https://dev.to/",
  "og:title": "DEV Community",
  "og:image": "https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8lvvnvil0m75nw7yi6iz.jpg",
  "og:description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "og:site_name": "DEV Community",
  "twitter:site": "@thepracticaldev",
  "twitter:title": "DEV Community",
  "twitter:description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "twitter:image:src": "https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8lvvnvil0m75nw7yi6iz.jpg",
  "twitter:card": "summary_large_image",
  "viewport": "width=device-width, initial-scale=1.0, viewport-fit=cover",
  "apple-mobile-web-app-title": "dev.to",
  "application-name": "dev.to",
  "theme-color": "#000000",
  "forem:name": "DEV Community",
  "forem:logo": "https://media.dev.to/cdn-cgi/image/width=512,height=,fit=scale-down,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F8j7kvp660rqzt99zui8e.png",
  "forem:domain": "dev.to",
  "title": "DEV Community"
}

pretty cool, isn't it?

Meta tags and why do we need them

Meta tags are HTML elements that are used to provide additional information about a page to search engines and other clients.
These tags typically include a name or property attribute that defines the type of information, and a content attribute that contains the value of that information. Here’s an example of two meta tags:

<meta name="description" content="The <meta> HTML element represents metadata that cannot be represented by other HTML meta-related elements, like <base>, <link>, <script>, <style> or <title>.">
<meta property="og:image" content="https://developer.mozilla.org/mdn-social-share.cd6c4a5a.png">

The first tag provides a description of the page, while the second is an Open Graph tag that defines an image to display when the page is shared on social media.

One practical application of meta tags is building a bookmark manager. Instead of manually adding the title, description, and image for each bookmark, you can automatically scrape this information from the bookmarked URL using meta tags.

Open Graph

Open Graph is an internet protocol that was originally created by Facebook to standardize the use of metadata within a webpage to represent the content of a page, it helps social networks generate rich link previews.
Read more about it here.

Why Deno?

1. Deno has secure defaults, meaning it requires explicit permission for file, network, and environment access, reducing the risk of security vulnerabilities.
2. Deno is built on web standards, uses ES Modules, and aims to use Web Platform APIs (like fetch) over proprietary APIs, making Deno code very similar to the code you'll write in the browser - but still has some spec deviation from the browser.
3. Deno has built-in TypeScript support, allowing you to write TypeScript code without a build step.
4. Deno comes with a standard library that includes modules for common tasks like HTTP servers, file system operations, and more.
5. Deno provides a Linter, Formatter and Test runner, allowing you to use the platform instead of relying on third party packages or tools, making it an all-in-one tool for Javascript development.
6. Deno provides Deno Deploy which is a scalable platform for serverless JavaScript/Typescript applications that are globally distributed, ensuring minimal latency and maximum uptime.

The API that we're building will consist of two parts, a function for fetching and parsing the meta tags, and an API server which will respond to HTTP requests.

Getting the meta tags

Let's start by going to Deno Deploy and signing in.
After signing in click on "New Playground"

This we'll give us a hello world starting point.
Now we'll add function that is called getMetaTags that accepts a url and uses the Fetch API to get the HTML of the requested URL and passes it on to a package for HTML parsing (deno-dom).
To add deno-dom to our project we can use the jsr package manager:

curl https://metatags.deno.dev/api/meta?url=https://dev.to

Now we'll use the Fetch API to get the HTML as text:

{
  "last-updated": "2024-10-15 15:10:02 UTC",
  "user-signed-in": "false",
  "head-cached-at": "1719685934",
  "environment": "production",
  "description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "keywords": "software development, engineering, rails, javascript, ruby",
  "og:type": "website",
  "og:url": "https://dev.to/",
  "og:title": "DEV Community",
  "og:image": "https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8lvvnvil0m75nw7yi6iz.jpg",
  "og:description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "og:site_name": "DEV Community",
  "twitter:site": "@thepracticaldev",
  "twitter:title": "DEV Community",
  "twitter:description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "twitter:image:src": "https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8lvvnvil0m75nw7yi6iz.jpg",
  "twitter:card": "summary_large_image",
  "viewport": "width=device-width, initial-scale=1.0, viewport-fit=cover",
  "apple-mobile-web-app-title": "dev.to",
  "application-name": "dev.to",
  "theme-color": "#000000",
  "forem:name": "DEV Community",
  "forem:logo": "https://media.dev.to/cdn-cgi/image/width=512,height=,fit=scale-down,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F8j7kvp660rqzt99zui8e.png",
  "forem:domain": "dev.to",
  "title": "DEV Community"
}

After getting the HTML, we can use deno-dom to parse it and then use standard DOM functions like querySelectorAll to get all the meta HTML elements, iterate on them and use getAttribute to get the name, property and content of each one of those tags:

<meta name="description" content="The <meta> HTML element represents metadata that cannot be represented by other HTML meta-related elements, like <base>, <link>, <script>, <style> or <title>.">
<meta property="og:image" content="https://developer.mozilla.org/mdn-social-share.cd6c4a5a.png">

Finally, we'll also query the