Beginner's Guide to Prevent Blog Content Crawling in WordPress

Are you looking for a way to prevent spammers and scammers from stealing your WordPress blog posts using content crawlers?

As a website owner, it is very frustrating to see someone steal your content without permission, monetize it, rank above you in Google, and steal your audience.

In this article, we will introduce what blog content crawl is, how to reduce and prevent content crawl, and even how to use the content crawler to benefit yourself.

What is blog content crawling in WordPress?

Blog content crawling is when you get content from multiple sources and repost it on another website. Usually, this is done automatically via the blog's RSS feed.

Unfortunately, it is very easy and very common to stolen your WordPress blog content in this way. If it happens to you, then you will understand how stressful and frustrating it is.

Sometimes your content will be simply copied and pasted directly to another website, including your formats, images, videos, etc.

Other times, your content will be republished without your permission, with the source and links back to your website. While this can help with your search engine optimization, you may just want to keep the original content on your website.

Why do content crawlers steal content?

Some of our users asked us why crawlers steal content. Usually, the main motivation for content theft is to profit from your hard work:

• Affiliate Commission: A dishonest affiliate marketer may use your content to drive traffic to their website through search engines to promote their niche products.
• Potential Client Development: Attorneys and real estate agents may spend money to ask people to add content and gain authority in the community, without realizing that this content is being stolen from other sources.
• Advertising revenue: Blog owners may crawl content to create knowledge centers in a specific area “for the benefit of the community” and then post ads on the website.

Is it possible to prevent content crawling completely?

In this article, we will show you some steps you can take to reduce and prevent content crawling. But unfortunately, there is no way to completely stop the stubborn thieves.

That's why we cover how to use content crawlers in the last section of this post. While you can’t always stop thieves, you may be able to get some traffic and revenue from the content they steal from you.

What should you do when you find someone copying your content?

Since it is impossible to block crawlers completely, you may one day find someone using what they stole from your blog. You may be wondering what to do when this happens.

Here are some of the ways people take when dealing with content crawlers:

• Do nothing: You may spend a lot of time fighting crawlers, so some popular bloggers decide to do nothing. Google has regarded well-known websites as authoritative, but not for smaller sites. Therefore, we don't think this approach is always the best.
• Delete: You can contact the crawler and ask them to delete the content. If they refuse, then you will submit a deletion notification. You can learn how to easily find and delete stolen content in WordPress in our guide.
• Take advantage: While we are actively working to remove content crawled from WPBeginner, we also use some techniques to get traffic and make money from crawlers. You can learn how to do it in the “Use Content Crawler” section below.

That being said, let's take a look at how to prevent blog crawling in WordPress. Since this is a comprehensive guide, we provide a directory for easy navigation:

1. Copyright or trademark of your blog name and logo
2. Make your RSS feed harder to crawl
3. Disable Trackback and Pingback
4. Block crawlers from accessing your WordPress website
5. Prevent images from being stolen in WordPress
6. Prevent your content from manually copying
7. Utilize content crawler

1. Copyright or trademark of the blog name and logo

Trademark and copyright laws protect your intellectual property, brand and business from many legal challenges. This includes the illegal use of your copyrighted material or your brand name and logo.

You should clearly display the copyright notice on your website. While your content is automatically protected by copyright law, displaying notifications will let you know that your content is copyrighted and that they cannot use your protected property for commercial purposes.

For example, you can add a copyright notice with a dynamic date to the WordPress footer. This will keep your copyright notice up to date.

This may prevent some users from stealing it. It will also help if you do need to send a stop letter or submit a DCMA complaint to delete the stolen content.

You can also apply for copyright registration online. The process can be complex, but luckily there are low-cost legal services that can help small businesses and individuals.

Learn how to register trademarks and copyrights for your blog name and logo in our guide.

2. Make your RSS feed harder to crawl

Since blog content crawling is usually done automatically through the blog's RSS source, let's take a look at some useful changes that can be made to the source.

Don't include full post content in WordPress RSS feeds

You can include only a summary of each article in your RSS feed, rather than the full content. This includes excerpts as well as post metadata such as dates, authors, and categories.

There is certainly a debate in the blog community about whether to have a full RSS feed or summary feed. We won't discuss this now, just want to say that one of the advantages of only summaries is that it helps prevent content crawling.

You can change settings by going to Settings»Read in the WordPress admin panel. You need to select the "Excerpt" option and click the "Save Changes" button.

Now, the RSS feed will display only excerpts of your article. If someone steals your content through your RSS feed, they will only get a summary, not a full post.

If you want to adjust the summary, you can check out our guide on how to customize WordPress excerpts.

Optimize your RSS feed to prevent crawling

There are other ways you can optimize your WordPress RSS feed to protect your content, get more backlinks, increase network traffic, and more. One of the best ways is to delay posting appearing in the RSS feed.

The benefit of this is that when you delay posting appearing in your RSS feed, you can give search engines time to crawl and index your content before it appears elsewhere, such as a crawler website. Search engines will then consider your website as an authoritative website.

The safest and easiest way is to use WPCode because it has a way to automatically add the correct custom code to WordPress.

For detailed instructions, see a guide on how to delay posting appear in WordPress RSS feeds.

3. Disable Trackbacks, Pingbacks, and REST APIs

In the early days of blogging, citation quotes and pingback were a way for blogs to notify links to each other. When someone links to a post on your blog, their website will automatically send pings to your website.

This pingback will then appear in your blog's Comment Review queue with a link to its website. If you approve, they will get backlinks and mentions from your website.

This inspires spammers to crawl your website and send references. Fortunately, you can disable trackback and pingback, reducing the reason crawlers can steal your content.

For more information, check out our guide on disabling citations for all future posts. You may also want to learn how to disable quotes and pings for existing WordPress posts.

Disable WordPress REST API

In addition to trackback and pingback, we recommend disabling the WordPress REST API as it makes it easier for spammers to crawl your content.

We have a detailed guide on how to disable the WordPress REST API.

All you need to do is install and activate the free WPCode plugin and use its pre-made fragments to disable the REST API.

4. Block crawlers from accessing your WordPress website

One way to prevent crawlers from stealing your content is to cancel their access to your website. You can do this manually by blocking its IP address, but most users will find it easier to use security plugins, such as web application firewalls.

Block crawlers with security plugins (recommended)

Manually blocking scrapers is very tricky and requires a lot of work. Especially because many hacker attempts and attacks are performed using various random IP addresses from around the world. It is nearly impossible to keep up with all these random IP addresses.

This is why you need a Web Application Firewall (WAF) such as Wordfence or Securi. They act as a barrier between your website and all incoming traffic by monitoring your website traffic and blocking them before common security threats reach your WordPress site.

For the WPBeginner website, we use Sucuri. It is a website security service that protects your website from such attacks using the website application firewall.

Basically, all your website traffic goes through a secure service server and checks for suspicious activity. They will automatically block suspicious IP addresses from fully accessing your website. Learn how Sucuri helps us block 450,000 WordPress attacks in 3 months.

Manually block or redirect the crawler's IP address

Advanced users may also want to manually block the crawler's IP address. This requires more work, but once you understand the address of the crawler, you can target it specifically. Web developer Jeff Star recommends this approach when writing articles on how to handle content crawlers.

Note: Adding code to website files can be dangerous. Even a small mistake can cause a major error on your website. That's why we only recommend this method to advanced users.

You can find the crawler's IP address by accessing the Original Access Log in the cPanel dashboard for your WordPress hosting account. You need to look for IP addresses with unusually high request counts and log them, such as copying them into a separate text file.

Tip: You need to make sure that you will not end up blocking your own, legitimate users, or search engines from accessing your website. Copy a suspicious IP address and use the online IP lookup tool to learn more.

Once you are sure that the IP address belongs to the crawler tool, you can block it using the cPanel "IP Blocker" tool or by adding the following code to the root .htaccess file:

The best part is that these banners will also appear on the crawler website.

In our case, we always add some disclaimer at the bottom of the post of the RSS feed. By doing so, we can get backlinks to the original article from the crawler website.

This lets Google and other search engines know that we are authoritative. It also lets their users know that the website is stealing our content.

For more tips, check out our guide on how to control RSS source footer in WordPress.

We hope this tutorial helps you understand how to prevent blog content from being crawled in WordPress. You may also want to check out our ultimate WordPress security guide or expert selection of the best WordPress analytics solutions.

The above is the detailed content of Beginner's Guide to Prevent Blog Content Crawling in WordPress. For more information, please follow other related articles on the PHP Chinese website!