WordPress at a glance

do_robots() WP 2.1.0

Displays the robots.txt file content. Sets appropriate HTTP headers. Used for creation of dynamic robots.txt file.

In WP, this function is attached to do_robots hook, which is triggers in template-loader.php file for /robots.txt request:

// Process feeds and trackbacks even if not using themes.
if ( is_robots() ) :
	 * Fired when the template loader determines a robots.txt request.
	 * @since 2.1.0
	do_action( 'do_robots' );
elseif ( is_feed() ) :

The result of this function can be changed with the hooks:

  • do_robotstxt — the action triggers at the beginning of the function. We can add any text on this hook and it will be displayed before the basic robots.txt content (code).
  • robots_txt — the filter triggers at the end of this function and filter all robots.txt content (code).

You can read more about robots.txt in my article.

The function sends HTTP headers, so using it after the HTTP headers have already been sent will cause an error.

It's better to use this function to create robots.txt file, but not create the file physically in the site root. Because it gives the opportunity for plugins to modify the content.

Hooks from the function

Null. Nothing.


#1 Create dynamic robots.txt

Add this code in functions.php (in this case we don't need to create a physical robots.txt file):

add_action( 'do_robotstxt', 'my_robotstxt' );
function my_robotstxt(){

	$lines = [
		'User-agent: *',
		'Disallow: /wp-admin/',
		'Disallow: /wp-includes/',

	echo implode( "\r\n", $lines );

	die; // обрываем работу PHP

Now, when going to http://example.com/robots.txt, the following text will be displayed:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

#2 Modify robots.txt content using robots_txt filter

Let's add "Disallow: */comments" rule.

add_filter('robots_txt', 'add_robotstxt');
function add_robotstxt($text){
	$text .= "Disallow: */comments";
	return $text;

So now http://example.com/robots.txt contains the following text:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: */comments


Since 2.1.0 Introduced.
Since 5.3.0 Remove the "Disallow: /" output if search engine visiblity is discouraged in favor of robots meta HTML tag in wp_no_robots().

Code of do_robots() WP 5.5.1

function do_robots() {
	header( 'Content-Type: text/plain; charset=utf-8' );

	 * Fires when displaying the robots.txt file.
	 * @since 2.1.0
	do_action( 'do_robotstxt' );

	$output = "User-agent: *\n";
	$public = get_option( 'blog_public' );

	$site_url = parse_url( site_url() );
	$path     = ( ! empty( $site_url['path'] ) ) ? $site_url['path'] : '';
	$output  .= "Disallow: $path/wp-admin/\n";
	$output  .= "Allow: $path/wp-admin/admin-ajax.php\n";

	 * Filters the robots.txt output.
	 * @since 3.0.0
	 * @param string $output The robots.txt output.
	 * @param bool   $public Whether the site is considered "public".
	echo apply_filters( 'robots_txt', $output, $public );

From category: Uncategorized

vladlu 100vlad.lu
Editors: kama 100
No comments
    Log In . Register