do_robots() │ WP 2.1.0

Displays the robots.txt file content. Sets appropriate HTTP headers. Used for creation of dynamic robots.txt file.

In WP, this function is attached to do_robots hook, which is triggers in template-loader.php file for /robots.txt request:

// Process feeds and trackbacks even if not using themes.
if ( is_robots() ) :
	/**
	 * Fired when the template loader determines a robots.txt request.
	 *
	 * @since 2.1.0
	 */
	do_action( 'do_robots' );
	return;
elseif ( is_feed() ) :
	...

The result of this function can be changed with the hooks:

  • do_robotstxt — the action triggers at the beginning of the function. We can add any text on this hook and it will be displayed before the basic robots.txt content (code).
  • robots_txt — the filter triggers at the end of this function and filter all robots.txt content (code).

You can read more about robots.txt in my article.

The function sends HTTP headers, so using it after the HTTP headers have already been sent will cause an error.

It's better to use this function to create robots.txt file, but not create the file physically in the site root. Because it gives the opportunity for plugins to modify the content.

Hooks from the function

Return

null. Nothing (null).

Examples

1

#1 Modify robots.txt content using robots_txt filter

Let's add "Disallow: */comments" rule. 

add_filter('robots_txt', 'add_robotstxt');
function add_robotstxt($text){
	$text .= "Disallow: */comments";
	return $text;
}

So now http://example.com/robots.txt contains the following text:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: */comments
0

#2 Create dynamic robots.txt

Add this code in functions.php (in this case we don't need to create a physical robots.txt file):

add_action( 'do_robotstxt', 'my_robotstxt' );
function my_robotstxt(){

	$lines = [
		'User-agent: *',
		'Disallow: /wp-admin/',
		'Disallow: /wp-includes/',
		'',
	];

	echo implode( "\r\n", $lines );

	die; // обрываем работу PHP
}

Now, when going to http://example.com/robots.txt, the following text will be displayed:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Changelog

Since 2.1.0 Introduced.
Since 5.3.0 Remove the "Disallow: /" output if search engine visibility is discouraged in favor of robots meta HTML tag via wp_robots_no_robots() filter callback.

do_robots() code WP 6.7.1

wp-includes/functions.php 
function do_robots() {
	header( 'Content-Type: text/plain; charset=utf-8' );

	/**
	 * Fires when displaying the robots.txt file.
	 *
	 * @since 2.1.0
	 */
	do_action( 'do_robotstxt' );

	$output = "User-agent: *\n";
	$public = get_option( 'blog_public' );

	$site_url = parse_url( site_url() );
	$path     = ( ! empty( $site_url['path'] ) ) ? $site_url['path'] : '';
	$output  .= "Disallow: $path/wp-admin/\n";
	$output  .= "Allow: $path/wp-admin/admin-ajax.php\n";

	/**
	 * Filters the robots.txt output.
	 *
	 * @since 3.0.0
	 *
	 * @param string $output The robots.txt output.
	 * @param bool   $public Whether the site is considered "public".
	 */
	echo apply_filters( 'robots_txt', $output, $public );
}

Uncategorized