Robots API

The Robots API appeared in WordPress 5.7. It is needed for centralized modification of the meta-field "robots" directives, so that themes and plugins do not create their own "bicycles", which may conflict with each other. Now everything is done through the wp_robots filter.

About the robots meta-field

The robots meta-field allows specifying in detail how the page should be processed by the robot - how it should be indexed and what aspects should be conveyed in the search results for users.

The robots meta-field should be placed in the <head> section of the page:

<!DOCTYPE html>
<html>
	<head>
		<meta name="robots" content="max-image-preview:large" />
	</head>
...

All functions related to the new Robots API are located in the file /wp-includes/robots-template.php.

Also, see possible directives of the robots meta-field:

For Google.
For Yandex.

Function `wp_robots()`

In WP 5.7, the function wp_robots() appeared, which outputs the robots meta-tag.

the function is automatically called by the WordPress core on all front-end pages, this happens on the wp_head event. Thus, the "robots" meta-tag will be automatically output on all front-end pages if at least one robots directive is specified.

The function is "hooked" in the file /wp-includes/default-filters.php:

add_action( 'wp_head', 'wp_robots', 1 );

Plugins or themes should not call wp_robots() separately! This may be necessary in special cases, for example, when the template cannot use wp_head(). In this case, the function should be attached to its own filter or called directly in the code. Example:

Attaching to its own filter:

add_action( 'my_custom_template_head', 'wp_robots' );

Direct call:

<?php wp_robots() ?>

How to use Robots API

As mentioned earlier, the "robots" meta-tag is automatically output on all site pages (if the site is open for indexing in the settings) at the moment of the wp_head event. This means that there is no need to call any functions to add the "robots" meta-tag to the page, but it is necessary to simply add/remove directives through the wp_robots filter.

Examples

The wp_robots filter passes an array with pairs of [ directive => value ] to the attached function. That is, your directive is added with this code:

add_filter( 'wp_robots', 'my_wp_robots_directives' );

function my_wp_robots_directives( $robots ) {
	$robots['follow'] = true;

	return $robots;
}

As a result, the specified directive will be added to the existing ones:

<meta name="robots" content="max-image-preview:large, follow" />

To remove an existing directive, it needs to be removed from the array:

add_filter( 'wp_robots', 'my_wp_robots_directives' );

function my_wp_robots_directives( $robots ) {
	unset( $robots['max-image-preview'] );
	$robots['follow'] = true;

	return $robots;
}

We will get:

<meta name="robots" content="follow" />

Directive `max-image-preview:large`

Since WP 5.7, WordPress by default adds the directive max-image-preview:large to the "robots" meta-tag on all pages.

<meta name="robots" content="max-image-preview:large" />

To disable this directive, the following code can be used:

remove_filter( 'wp_robots', 'wp_robots_max_image_preview_large' );

max-image-preview: [setting]

This directive determines the maximum size of images that can be displayed in the search results for this page.

If the max-image-preview directive is not specified, a default image preview size will be possible.

Allowed values for [setting]:

none: No image for preview.
standard: Standard image preview can be shown by default.
large: A larger image can be shown, up to the maximum viewable area width.

This applies to all types of search results (Google web search, Google Images, recommendations, and data provided by Assistant). The restriction does not apply in cases where the publisher has provided separate permission to use the content, for example, added structured data or entered into a licensing agreement with Google. In particular, structured data can define the canonical and AMP versions of the article.

If you do not want your canonical pages and their AMP versions to be represented in Google Search and recommendations as enlarged icons, specify the value "standard" or "none" in the max-image-preview directive.

Other Default Directives Added by WP

The Robots API is also used in WP on some specific pages:

If the "noindex the site" option is enabled, the robots will include the directives noindex, nofollow (previously included as well) and the directive max-image-preview:large will not be added.
The Customizer preview adds the noindex directive (previously included).
The login page (wp-login.php) and (wp-activate.php) contain the directives noindex, noarchive, as well as <meta name="referrer" content="strict-origin-when-cross-origin" /> (previously included).
In multisite, the site activation page (wp-activate.php), where new users can confirm their created site, also contains the directives noindex, noarchive, as well as <meta name="referrer" content="strict-origin-when-cross-origin" /> (previously included).

Notes

Since August 2021, Google does not support the "follow" directive in the robots meta-tag.

In fact, nothing has changed for optimizers. Yandex still takes this directive into account.

Initially, a question was asked by a specialist:

"...there is no information in Google's documentation about the "follow" directive... Does Google now ignore this directive?"

To which John Mueller responded:

Translation: "Yes, this value is the default, so there is no need to specify it additionally. It's the same as adding the "Allow: /" directive as the only one in the robots.txt file. Or like making a road and putting a sign "cars allowed". You won't surprise anyone with that."

Deprecated WP PHP Functions

The new Robots API replaces various separately introduced robots meta-tags that the WordPress core used before version 5.7. Thus, the following functions are deprecated and should no longer be used anywhere:

Instead of noindex(), wp_robots_noindex() appeared, it is attached to the "wp_robots" filter.
Instead of wp_no_robots(), wp_robots_no_robots() appeared, also attached to the "wp_robots" filter.
Instead of wp_sensitive_page_meta(), which created 2 meta-tags, two separate functions appeared:
- For the noindex, noarchive directives, the function wp_robots_sensitive_page() appeared, it should be attached to the wp_robots hook where necessary.
- For the additional referrer meta-tag, the function wp_strict_cross_origin_referrer() appeared, it needs to be attached to the wp_head hook.

All functions by default attached to the "wp_robots" hook in the core are in the file /wp-includes/default-filters.php:

// Robots filters.
add_filter( 'wp_robots', 'wp_robots_noindex' );
add_filter( 'wp_robots', 'wp_robots_noindex_embeds' );
add_filter( 'wp_robots', 'wp_robots_noindex_search' );
add_filter( 'wp_robots', 'wp_robots_max_image_preview_large' );

if ( isset( $_GET['replytocom'] ) ) {
	add_filter( 'wp_robots', 'wp_robots_no_robots' );
}