Robots-txt-generator Freeware

rate ME

Developer

 Robots txt

Version:

 1.01

Size in Kb

 82

Downloads

 9

Our rating

 

Popularity

The spiders that visit your site consumes much of your bandwidth?If you want to control the robots or spiders that visit your site, robots-txt generator is the solution.It is very common that the webmaster of each site spends a lot of money paying for bandwidth and it is unfair that a significant portion of that bandwidth is consumed spiders.Robots txt generator, you can stop or control the bots.You can see tutorial-and-guides here :http://robots-txt.info/category/robotstxt-tutorial-and-guides/Some robots or spiders saturate your server making countless queries per minute?With Robots txt generator, you can limit how many requests can make any robot or spider per hour or per minute.In addition, the program has a database of more than 300 bots, allowing you to search on the database the source and owner of any bot you have found in your logs.You can even specify different restrictions for different bots. Per example, you can allow google bot, but disallow google-image-bot. You can specify a different restriction level for each one of the 300 bots in the database or can specify some restrition to the “general” bots, plus some specific for some specifi bots. Like giving a ban to some specific bot.. Robots.txt Info.

Platforms

WinXP, Win2003, Win2000, Windows Vista

More Windows 8 Downloads Related to Robots-txt-generator

 

PJN Sitemap Generator will crawl your entire website up to 6 levels deep.The user types in the home page link, like http://www.pjncomputersolutions.com then click "go".The sitemap generator will crawl from your home page and will list every internal link in the left column and every ... Size: 10k, Revised: September 16th 2009, By: PJN Computer Solutions

Complete Domain Name generator and availability checker software finds you the right domain names. Enter keyword(s) and the software suggests and checks availability of multiple versions of those keywords in various combinations, with and without common pre-fixes and suffixes attached (e.g. "eTractors.com" instead of "tractors.com"). ... Size: 506k, Revised: April 8th 2009, By: HostingDude.com

Complete Domain Name generator and availability checker software finds you the right domain names. Enter keyword(s) and the software suggests and checks availability of multiple versions of those keywords in various combinations, with and without common pre-fixes and suffixes attached (e.g. "eTractors.com" instead of "tractors.com"). ... Size: 506k, Revised: May 12th 2008, By: HostingDude.com

Site Map Builder allows you to create the Google SiteMaps XML needed within your root directory in order to provide information about your site to Google. This XML must have some syntax that SiteMapBuilder.NET will generate for you Extract links from a site recursively Extract ... Size: 1,393k, Revised: October 17th 2008, By: SiteMapBuilder.NET

Keyword Research Software that Help you Build A Huge Profitable Keyword List Instantly. Analyze the demand and supply of any niche market and saves you time required for the necessary research. Uncover hidden niche markets with a push of a button. Extract the no. of ... Size: 1,843k, Revised: August 1st 2009, By: 4Bec Internet Inc.

Blog Finder Software that Help you Boost Your Link Building & SEO Efforts.Identifies Wordpress blogs with the 'nofollow' tag in the comments field in ANY niche you choose. Saves you time and effort from searching for blogs manually to leave comments on and building backlinks. ... Size: 1,823k, Revised: July 29th 2009, By: 4Bec Internet Inc.

Meta tags analyzer and generator. Use this tool too analyze your website to get top result in search engines. Size:  , Revised: September 25th 2012, By: lauhost.com

Need a good domain name and still searching for one by one? Avensen Domain Name Finder is a domain name generation and availability check software, capable to lookup thousands of domain names per minute. Domain names can be generated from keyword groups, entered manually or ... Size: 1,894k, Revised: August 30th 2012, By: Avensen Software

Visual Web Spider is a fully automated, multithreaded web crawler that allows you to collect and index specific web pages on the Internet.Tired of surfing the Internet searching for relevant web pages? Wouldn't it be nice if you had a personal, customizable web crawler that ... Size:  , Revised: September 16th 2012, By: Newprosoft.com

IDwhois is a free, small and efficient whois client running under Windows system that lets you retrieve registration information for domain names. IDwhois is using the whois TCP-based query/response protocol which is widely used for querying a database in order to determine the owner of ... Size: 553k, Revised: October 17th 2008, By: IDwhois


User Reviews
 
Name: *
E-mail: *
Text: *
Verification Code: *
Notice: * - required fields