bootstrap/site/layouts/robots.txt

# www.robotstxt.org

{{- $allowCrawling := false -}}

{{ if $allowCrawling }}
# Allow crawling of all content
{{- end }}
User-agent: *
Disallow:{{ if not $allowCrawling }} /{{ end }}
Sitemap: {{ "/sitemap.xml" | absURL }}
Switch to Hugo. No more Ruby dependency and most important more than 10 times less build time (at least on Windows)! 2019-01-09 00:33:28 +08:00			`# www.robotstxt.org`

Prepare docs for v5.getbootstrap.com * use v5.getbootstrap.com as the site URL/CNAME * block robots from crawling * specify next npm tag 2020-06-17 02:17:28 +08:00			`{{- $allowCrawling := false -}}`
robots.txt: adapt for Netlify. (#29192) Since we build with `HUGO_ENV` set to `production` on Netlify, use another variable to prevent crawling. 2019-08-02 22:26:25 +08:00
			`{{ if $allowCrawling }}`
Switch to Hugo. No more Ruby dependency and most important more than 10 times less build time (at least on Windows)! 2019-01-09 00:33:28 +08:00			`# Allow crawling of all content`
			`{{- end }}`
			`User-agent: *`
robots.txt: adapt for Netlify. (#29192) Since we build with `HUGO_ENV` set to `production` on Netlify, use another variable to prevent crawling. 2019-08-02 22:26:25 +08:00			`Disallow:{{ if not $allowCrawling }} /{{ end }}`
Use `absURL` so that we get consistent URLs. (#29226) 2019-08-08 23:30:43 +08:00			`Sitemap: {{ "/sitemap.xml" \| absURL }}`