[vc_row unlock_row_content=”yes” row_height_percent=”0″ override_padding=”yes” h_padding=”0″ top_padding=”0″ bottom_padding=”0″ back_image=”103507″ back_repeat=”no-repeat” back_position=”center center” overlay_alpha=”0″ gutter_size=”3″ column_width_percent=”0″ shift_y=”0″ z_index=”0″ uncode_shortcode_id=”550980″ back_size=”cover” el_class=”inner-banner-slide amp-banner-section”][vc_column column_width_percent=”100″ gutter_size=”3″ back_image=”103506″ back_repeat=”no-repeat” back_position=”center center” overlay_alpha=”0″ shift_x=”0″ shift_y=”0″ shift_y_down=”0″ z_index=”0″ medium_width=”0″ mobile_width=”0″ width=”1/1″ uncode_shortcode_id=”180080″ back_size=”cover”][vc_row_inner row_inner_height_percent=”0″ overlay_alpha=”0″ gutter_size=”3″ shift_y=”0″ z_index=”0″ uncode_shortcode_id=”164561″ el_class=”home-banner-container” css=”.vc_custom_1641456384377{padding-right: 15px !important;padding-left: 15px !important;}”][vc_column_inner width=”2/3″][vc_custom_heading text_color=”color-xsdn” heading_semantic=”h1″ text_font=”font-555555″ text_size=”fontsize-402681″ uncode_shortcode_id=”209155″ text_color_type=”uncode-palette”]
Robots.txt
[/vc_custom_heading][vc_column_text uncode_shortcode_id=”337346″ css=”.vc_custom_1656661234560{margin-top: 10px !important;}”][/vc_column_text][/vc_column_inner][vc_column_inner width=”1/3″][/vc_column_inner][/vc_row_inner][/vc_column][/vc_row][vc_section overlay_alpha=”50″ uncode_shortcode_id=”162784″ el_class=”amp-content-section”][vc_row][vc_column width=”1/2″][vc_single_image media=”108479″ media_width_percent=”100″ uncode_shortcode_id=”459978″][/vc_column][vc_column width=”1/2″][vc_column_text uncode_shortcode_id=”675736″]The robot.txt represents computer data. More specifically, it is a text file that is used at the website level to indicate to crawlers (mainly those of search engines) how they should explore the pages of websites.
More technically, they are part of an exclusion protocol and represent the standards that govern how crawlers browse the web, and access and index content.
It is included in the root of your website so that if the address of your site is: www.mysite.com, the URL leading to the robots.txt is:
- www.mysite.com/robots.txt
[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column width=”1/1″][vc_column_text uncode_shortcode_id=”271785″]
References –
- https://developers.google.com/search/docs/advanced/robots/intro
- https://en.wikipedia.org/wiki/Robots_exclusion_standard
- https://moz.com/learn/seo/robotstxt
Other SEO Terminology
Google Search Console | Disallow | Allow Directive | Crawl[/vc_column_text][/vc_column][/vc_row][vc_row row_height_percent=”0″ back_color=”T-Red-155595″ overlay_alpha=”50″ gutter_size=”3″ column_width_percent=”100″ shift_y=”0″ z_index=”0″ uncode_shortcode_id=”115159″ back_color_type=”uncode-palette”][vc_column width=”1/1″][vc_column_text uncode_shortcode_id=”145216″][loc_shortcode][/vc_column_text][/vc_column][/vc_row][/vc_section][vc_row][vc_column width=”1/1″][vc_custom_heading uncode_shortcode_id=”203151″]Knowledge Base Articles[/vc_custom_heading][uncode_index el_id=”index-189527″ isotope_mode=”fitRows” loop=”size:3|order_by:rand|post_type:post|taxonomy_count:10″ screen_lg=”1000″ screen_md=”600″ screen_sm=”480″ gutter_size=”3″ post_items=”media|featured|onpost|original,title,text|excerpt,sep-one|full” single_overlay_opacity=”50″ single_padding=”2″ uncode_shortcode_id=”672360″][/vc_column][/vc_row]
