bootstrap/site/layouts/robots.txt

# www.robotstxt.org

{{- $isNetlify := eq (getenv "NETLIFY") "true" -}}
{{- $allowCrawling := and (not $isNetlify) hugo.IsProduction -}}

{{ if $allowCrawling }}
# Allow crawling of all content
{{- end }}
User-agent: *
Disallow:{{ if not $allowCrawling }} /{{ end }}
Sitemap: {{ "/sitemap.xml" | absURL }}
Switch to Hugo. No more Ruby dependency and most important more than 10 times less build time (at least on Windows)! 2019-01-09 00:33:28 +08:00			`# www.robotstxt.org`

Revert v5.getbootstrap.com changes 2020-11-25 03:05:11 +08:00			`{{- $isNetlify := eq (getenv "NETLIFY") "true" -}}`
Switch to using `hugo.IsProduction` (#39763) 2024-03-08 18:41:17 +08:00			`{{- $allowCrawling := and (not $isNetlify) hugo.IsProduction -}}`
robots.txt: adapt for Netlify. (#29192) Since we build with `HUGO_ENV` set to `production` on Netlify, use another variable to prevent crawling. 2019-08-02 22:26:25 +08:00
			`{{ if $allowCrawling }}`
Switch to Hugo. No more Ruby dependency and most important more than 10 times less build time (at least on Windows)! 2019-01-09 00:33:28 +08:00			`# Allow crawling of all content`
			`{{- end }}`
			`User-agent: *`
robots.txt: adapt for Netlify. (#29192) Since we build with `HUGO_ENV` set to `production` on Netlify, use another variable to prevent crawling. 2019-08-02 22:26:25 +08:00			`Disallow:{{ if not $allowCrawling }} /{{ end }}`
Use `absURL` so that we get consistent URLs. (#29226) 2019-08-08 23:30:43 +08:00			`Sitemap: {{ "/sitemap.xml" \| absURL }}`