robots_txt │ filter-hook │ WP 3.0.0

Allows you to append content to the already generated /robots.txt page.

By default, WP 5.5 creates the following content for /robots.txt:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: http://example.com/wp-sitemap.xml

This hook lets you append data to the existing robots.txt content. The code can be placed in the theme's functions.php file.

Usage

add_filter( 'robots_txt', 'wp_kama_robots_txt_filter', 10, 2 );

/**
 * Function for `robots_txt` filter-hook.
 * 
 * @param string $output The robots.txt output.
 * @param bool   $public Whether the site is considered "public".
 *
 * @return string
 */
function wp_kama_robots_txt_filter( $output, $public ){

	// filter...
	return $output;
}

$output(string)

Original robots.txt content.

$public(bool)

Whether the site is open to indexing, based on the "Discourage search engines from indexing this site" Reading setting.

Examples

#1 Add custom directives to robots.txt

A simple example that disallows the /private/ directory.

add_filter( 'robots_txt', 'append_private_disallow', 10, 2 );

function append_private_disallow( $output, $public ) {
	$output .= "\nDisallow: /private/\n";
	return $output;
}

#2 Another example of appending to robots_txt

// Append to the basic robots.txt.
// -1 means before wp-sitemap.xml.
add_action( 'robots_txt', 'wp_kama_robots_txt_append', -1 );

function wp_kama_robots_txt_append( $output ){
	$str = <<<'TXT'
		Disallow: /cgi-bin             # Standard hosting directory.
		Disallow: /?                   # All query parameters on the home page.
		Disallow: *?s=                 # Search.
		Disallow: *&s=                 # Search.
		Disallow: /search              # Search.
		Disallow: /author/             # Author archive.
		Disallow: */embed              # All embeds.
		Disallow: */page/              # All types of pagination.
		Disallow: */xmlrpc.php         # WordPress API file.
		Disallow: *utm*=               # URLs with UTM tags.
		Disallow: *openstat=           # URLs with OpenStat tags.
		TXT;

	$str = trim( $str );
	$str = preg_replace( '/^[\t ]+(?!#)/mU', '', $str );
	$output .= "$str\n";

	return $output;
}

Now open /robots.txt to see:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cgi-bin             # Standard hosting directory.
Disallow: /?                   # All query parameters on the home page.
Disallow: *?s=                 # Search.
Disallow: *&s=                 # Search.
Disallow: /search              # Search.
Disallow: /author/             # Author archive.
Disallow: */embed              # All embeds.
Disallow: */page/              # All types of pagination.
Disallow: */xmlrpc.php         # WordPress API file.
Disallow: *utm*=               # URLs with UTM tags.
Disallow: *openstat=           # URLs with OpenStat tags.

Sitemap: http://example.com/wp-sitemap.xml