A robots.txt file is an element of your website read by search engine crawlers that identifies portions of your site that you don’t want indexed by search engines. Robots.txt is also called the Robots Exclusion Protocol. Website owners generally want search engine robots to crawl and index their sites. However, there may be reasons why, as a website owner, you don’t want your entire site indexed. That’s where a robots.txt file comes in handy. It tells the web crawlers which pages to leave out of the index.

Each LiveEdit website comes with a predetermined list of pages that are excluded from indexing. For example, you have a website administration folder as part of your website structure. There’s no value in having that content indexed, so the robots.txt file is a way of telling the search engines to ignore these pages.

Hiding an In-Progress Website: If a page on your site (or your entire site) is not ready to show off to the world, it can be excluded from the search engine results.

1. Log in to your website.

2. Navigate to the Account Settings page by hovering over your name in the top right hand corner of your screen and clicking on Account Settings.

3. Click on the 'Manage Your Aliases' tab.

4. Click on the pencil and paper icon to the right of the first alias (website url) that you would like to block from the SERPs.


 5. In the Edit box that pops up, change the status of the robots.txt file to 'Deny All.'


6. Repeat Steps 4 & 5 for any additional aliases that you would like to deny.


Hiding an In-Progress Page: If a single page on your site is not ready to show off to the world, simply add its url to the robots.txt file to keep it hidden from search bots. (Just remember to remove the block form the robots.txt file when it's time to launch the page!)

1. Log in to your website.

2. Navigate to the Account Settings page by hovering over your name in the top right hand corner of your screen and clicking on Account Settings.

 

3. In the menu beside 'robots.txt,' choose 'Advanced.'


If you are a MINDBODY User:                     If you are NOT a MINDBODY User:
Sitemap: http://YOURURL.com/sitemap.xml
User-agent: *
Disallow: /manage
Disallow: /admin
Disallow: /pages/emarketing-1
Disallow: /emarketing-1
Disallow: /pages/enewsletter-1
Disallow: /enewsletter-1
Disallow: /pages/enewsletter-2
Disallow: /enewsletter-2
Disallow: /service/profile/manage
Disallow: /service/access/manage
Disallow: /service/templates/manage
Disallow: /service/pages/manage
Disallow: /service/account/manage
Disallow: /service/form/manage
Disallow: /service/emailmarketing/manage
Disallow: /service/analytics/manage
Disallow: /service/blog/manage
Disallow: /service/sms/manage
Disallow: /service/calendar/manage
Disallow: /service/file/manage
Disallow: /*apps/mindbody/myaccount$
Disallow: /*apps/mindbody/checkout$
Disallow: /*apps/mindbody/list-schedule?forward=$
Disallow: /*apps/mindbody/list-schedule?back=$
Disallow: /*apps/mindbody/signup$
Sitemap: http://YOURURL.com/sitemap.xml
Disallow: /admin
Disallow: /*?view=mobile$
Disallow: /*?view=full$
Disallow: /pages/email-marketing
Disallow: /email-marketing
Disallow: /service/profile/manage
Disallow: /service/access/manage
Disallow: /service/templates/manage
Disallow: /service/pages/manage
Disallow: /service/account/manage
Disallow: /service/form/manage
Disallow: /service/emailmarketing/manage
Disallow: /service/analytics/manage
Disallow: /service/blog/manage
Disallow: /service/sms/manage
Disallow: /service/calendar/manage
Disallow: /service/file/manage

 

 

4. First copy the default robots.txt from the table above (for either MINDBODY users or non-MINDBODY users) and paste it into the text box in your account settings page. Edit the first line to reflect your website's url.


5. To add a page to the exclusion list in your robots.txt file, add Disallow: followed by its page url. For example, if my site is mysite.com and I want to exclude mysite.com/contact from the serps, I would add a the following to my robots.txt:  

Disallow: /contact


Guaranteed exclusion? – Standards-based web crawlers (Google, Bing/Yahoo!) and other well-known search engine robots look for and heed your robots.txt file instructions. Lesser-known web crawlers may not, so there’s no assurance that your identified pages won’t be indexed somewhere on the web.