professional robots.txt

Question

Hey All.
How should a professional robots.txt for seotoaster look like?
Which directories should be blocked?
/cache/ ??

Best regards

Ronny

Answer 1

Hello Ciatronical

For example you should use:

User-agent: *
Disallow: /system/
allow: /system/css
allow: /system/js
Disallow: /tmp/
allow: /tmp/*.js
allow: /tmp/*.css
Sitemap: https://_valid_domain.com_/sitemapindex.xml
User-agent: googlebot
Allow: /themes/
Allow: /system/js
Allow: /system/css

Regards, Pavel.

Comments

Thanks very much.

Best regards,
Ronny