zeus.ugent.be/content/robots.txt

User-agent: *
Disallow: /
Allow: /index.html
Allow: /$
Allow: /about/
Allow: /assets/images/
Allow: /blog/
Allow: /contact/
Allow: /events/
Allow: /feed.xml
Allow: /google6f2e77d0228abc35.html
Allow: /ical.ics
Allow: /projects/
Allow: /robots.txt
Allow: /sitemap.xml
Sitemap: https://zeus.gent/sitemap.xml
allow robots to do all the things 2017-05-29 12:44:07 +00:00			`User-agent: *`
Allow less crawling 2020-01-16 08:02:52 +00:00			`Disallow: /`
Allow crawling the home page 2020-01-16 14:50:17 +00:00			`Allow: /index.html`
Also allow plain home page 2020-01-17 18:31:11 +00:00			`Allow: /$`
Allow less crawling 2020-01-16 08:02:52 +00:00			`Allow: /about/`
Use sitemap to allow more pages 2020-01-16 08:24:05 +00:00			`Allow: /assets/images/`
			`Allow: /blog/`
Allow less crawling 2020-01-16 08:02:52 +00:00			`Allow: /contact/`
			`Allow: /events/`
Use sitemap to allow more pages 2020-01-16 08:24:05 +00:00			`Allow: /feed.xml`
			`Allow: /google6f2e77d0228abc35.html`
			`Allow: /ical.ics`
Allow less crawling 2020-01-16 08:02:52 +00:00			`Allow: /projects/`
Use sitemap to allow more pages 2020-01-16 08:24:05 +00:00			`Allow: /robots.txt`
			`Allow: /sitemap.xml`
			`Sitemap: https://zeus.gent/sitemap.xml`