# Robots.txt # Allow all robots to spider the domain User-agent: * Disallow: # Disallow directory /_assets/server/ User-agent: * Disallow: /_assets/server/ Disallow: /courses/ Disallow: /apply/ Disallow: /contact/ Disallow: /news/ Disallow: /policies/ Disallow: /payments/ Disallow: /about/