NOTE: This Sitemap Generator has been completely revamped and extremely easy to use!
All the features of our sitemap generator have been merged into our webmaster tool, simply click on the image to start. Below you will find instructions for creating XML Sitemaps, Google Sitemaps and HTML sitemaps.
* Before you jump into this, you should read everything on this page.
* The SITEMAP GENERATOR respects sessions! If you are logged into your site and have delete privileges, the sitemap generator will follow all links, including ‘delete links’, so play it safe and make sure you are NOT logged into your website!
* In order to Generate Sitemaps, please make sure you are running the latest version of java.
For other parts of the webmaster tool, such as checking your site for invalid links or finding out why it’s not ranking well, visit the Webmaster Tools page.
You’re probably here to generate a sitemap for your favorite search engine? At one time, Google had it’s own sitemap format, Yahoo had their own version and everyone else was trying to catch up. Fortunately, a standard was developed for XML Sitemaps which are now accepted by Google, Yahoo and others.
There is still the traditional html sitemap, and we have taken that into consideration when building the sitemap generator; our webmaster tool provides you with the option to generate a XML sitemap or HTML sitemap – how you export (save) the file is up to you.
Creative website owners have been using sitemaps to get their content recognized by search engines for years; these html sitemaps predate Google and XML site maps. The nice thing about this type of sitemap is that it can be read by humans AND search engines.
To create these older style sitemaps, website owners need only use our program to generate an HTML sitemap, upload that file to their server’s root directory (same location as the website’s main page) and place a link to that file in their footer (bottom of the website’s main page)
XML Sitemaps now the standard
A Google Sitemap (XML Sitemap) and the traditional sitemap differ in a number of ways. The XML sitemap is for made for search engines such as Yahoo, Google and MSN and when viewed by a human, it’s not pretty.
The traditional sitemap (HTML Sitemap) is a graphical way to present your website’s layout to humans to further aid in their navigation of your site.
If you want the search engines to see your webpages, a XML sitemap is the way to go. You can a Google sitemap (xml sitemap) and standard sitemap (html) running side by side and it will not negatively impact your site.
Why the Sitemap Generator
Google Sitemaps is a service that allows webmasters to submit an XML map of their site containing information such as change frequency, priority, date created and more. Google suggests a great sitemap program to create the site map, but you must have Python in order for it to run. For further reading on Google’s program, visit Google Sitemaps FAQ.
I needed a way to accomplish this without installing Python or extra software on my server and Java was the perfect solution!
Free Site Map Builder
This free and easy to use site map builder is fast, efficient and allows you to build sitemaps that can instantly be submitted to Google. The only requirement is a popular browser that supports java, such as Internet Explorer, Firefox and others.
Sitemap Generator Overview
The easy way to create a Google or XML Sitemap is by watching our 60 second Webmaster Tool Video.
It’s simple, just enter the website you would like to create a site map under the ‘settings’ tab, enter filters (if any), select any options (defaults work fine) and click the little green arrow to start crawling. The program will spider your site (nothing to do with spider bites:) and generate a sitemap using the default values, such as thread count of 5, include all files, etc. Care has been used to make sure the crawler does not follow external links :)
If you have a large site, you’ll need to increase the memory allocated to the webmaster tool, more about this can be found on our forum and this 60 second video on Increasing Java Memory.
Sitemap Generator Details
This site map generator (now part of the webmaster tool) is loaded with tons of features and consists of six tabs:
‘Project’ tab – allows you to save and load your sitemap project. This can be very handy when building XML Sitemaps (or any type of sitemap for that matter). An example might be that your site is a few million pages long in size and you have access to a computer for a short period of time. Simply stop the job, save it out and open it later. Note: The project saves out as an XML file – don’t confuse this with a XML Sitemap – they are NOT the same.
‘Settings’ tab – allows you to specify the way your site will be spidered and what will be included in your Google or XML sitemaps.
- Project Name – This will be the name of your project
- URL – The full address (including the http://) of the website you would like to create a Google or XML sitemap from
- Filters – you can tell our tool to include or exclude certain files when building your HTML, Google or XML sitemap. All filters are case sensitive, so if you are trying to exclude .Jpg files, then use *.Jpg, not *.jpg, etc.
- Include Filter – This is a list of path patterns, asterisk (*) wildcard supported, case sensitive. When the sitemap generator / crawler is about to process a website, it is validated against all inclusion patterns. If none have matched then the location will not be processed and will not be placed into sitemap. If you leave this area empty, then it is assumed that you want to include everything so that all locations will are processed (provided they conform to the rules listed below. Example:/dir1/* – process all files located below (recursively) directory “dir1″*.html – process all HTML files at any location.
You can also use Regular Expressions by prefixing the expression with the word “complex:”. For example, say that you would like to include all pages that start with http://[yoursite.com]/shopping/men/ and do not contain any numbers. You would use:
complex:http://[yoursite.com]/shopping/men/[a-zA-Z/]*$So, these would be included in the sitemap:
http://[yoursite.com]/shopping/men/trousers
http://[yoursite.com]/shopping/men/trousers/blue
but these would not be included:
http://[yoursite.com]/shopping/men/trousers/56
http://[yoursite.com]/shopping/men/trousers/blue1
- Exclude Filter – Patterns included here will be excluded from processing. Regular expressions accepted when prefixed with the word “complex:” When the sitemap generator crawls a website, that site is validated against exclusion patterns. If any patterns match, the location will not be processed and included into Google sitemap.Say that you have a directory located at http://[yoursite.com]/private/messages.php?voice=112 and you would like to prevent the messages.php from being included on the sitemap, then you would enter the following into the exclude filter*/private/messages*Want to exclude everything in the private directory from being included in the sitemap? Enter the following*/private/*
- Include Filter – This is a list of path patterns, asterisk (*) wildcard supported, case sensitive. When the sitemap generator / crawler is about to process a website, it is validated against all inclusion patterns. If none have matched then the location will not be processed and will not be placed into sitemap. If you leave this area empty, then it is assumed that you want to include everything so that all locations will are processed (provided they conform to the rules listed below. Example:/dir1/* – process all files located below (recursively) directory “dir1″*.html – process all HTML files at any location.
- Rules – Allows you to set rules when creating your XML, HTML or Google Sitemap.
- Load From – Provides you with the option to process files from the entire server and below the initial directory which is specified by URL parameter. For example, if the URL parameter is a server address, this option does not effect the behavior of the google sitemap generator; However, if you enter a directory, say for example http://www.popupcheck.com/news/index.html, only files below /news directory will be processed including any sub directories.
- Respect Robots.txt file – you can tell the sitemap generator to honor this file or to ignore it.
- Respect Meta Robots.txt – you can do as this meta tag instructs or ignore it.
- Respect No Follow – If the sitemap builder finds a link with a no-follow tag, it will ignore or follow it depending on your selection.
- Ignore invalid links – If you find links that try to back up past your root directory, then you can choose to not include this in your sitemaps.
- Exclude images – Check this. Images will not be included anyway (not used)
- Download only new files – this works when you have a sitemap project that you have saved out
- Options – Some cool stuff here and can be very important!
- Add skipped links to sitemap – when the webmaster tool crawls your site, it may find bad links. This option allows you to include those in the sitemap anyway.
- User agent – When you spider a site, you leave in the log files the name ‘AuditMyPC Webmaster Tool’ as the browser type. Some web hosting companies may block a browser if it is making too many requests. You can change the referrer (user agent) to something else by selecting from the drop down or typing it in!
- Max Level – the maximum depth the webmaster tool will dive down to when creating your sitemap. The default is all levels, but if you wanted to set it at 0, then only the root web pages would be found and included in your XML sitemap, if you have something like mysite.com/news/newat11.php then that file would be 1 deep and would not be included.
‘Crawler’ tab – Set the speed in which the sitemap is generated.
- Request Delay – Our XML sitemap generator works extremely fast, the downside of this is that some internet service providers may find this places a heavy load on the server. If this is the case, then you can place delays between requests.
- Connect Timeout – When building XML sitemaps which encounters pages that don’t load or take too long, a timeout can be set.
- Read Timeout – If the spider finds a page that goes on forever, you can specify a timeout for reading that page.
- Transfer Rate – Each thread can transfer web pages at a very fast rate. You can tone that down a bit if necessary, but the default works fine.
- Thread Count – The number of simultaneous crawling threads to run when creating the Google site map. This may significantly decrease overall crawling time if large number of threads are specified but will increase bandwidth usage – so use with caution or just run with the default.
- Once you click on the button, crawling will begin and you’ll be presented with status indicators for thread status, uri, values and more. All parameters are self explanatory and ‘Finished’ will appear once crawling is completed. You may stop the sitemap generation at any time by pressing the ‘cancel’ button.
‘Sitemap’ Tab – Contains a ton of information about each page and updates in real time.
all the locations / files that have been crawled. Under the ‘Sitemap’ tab, you have sub-tabs, such as ‘save sitemap’, ‘retry’, ‘row filter’, ‘column filter’ and ‘trees’.
- Export – this is where you decide what type of sitemap you would like to build.
You can choose ‘URL Raw List’, ‘Delimited’, ‘XML File (not for sitemaps)’, ‘Sitemap XML (For creating sitemaps for Google, Yahoo and others) ‘, ‘HTML Sitemap’ and a ‘HTML Report’. - Retry Failed – This option will retry to read pages from the sitemap that had problems on the last run
- Row Filter – When building a sitemap (crawling a site), you can filter out rows based on just about anything you can thing of (see the question mark next to each item for details).
- Column Filter – Save as the Row filter but for Columns.
- Find – This allows you to search your xml sitemap for text.
- You have the option to edit ‘Modified’, ‘Change frequency’ and ‘Priority’ cells for each row (or all rows – well get to that in a moment).
- Listing of URLs (pages) – You’ll see a listing of all your webpages that include items like Title, Status, Errors and more.
For the Google sitemap, you can set the ‘Change Frequency’ and ‘Change Priority’ for on or multiple urls by highlighting the desired url(s) and right clicking, then choosing your option. You can also delete page from your Google or XML Sitemap by simply highlighting the desired urls and pressing your computer’s delete key.- Change frequency – Tells Google Sitemaps the frequency that content of a particular URL will change. Your options are “always”, “hourly”, “daily”, “weekly”, “monthly”, “yearly” or “never”. The value “always” should be used to describe documents that change each time they are accessed. The value “never” should be used to describe archived URLs.
- Priority – The priority of a particular URL relative to other pages on your site. You may select between 0.0 and 1.0, where 0.0 identifies the lowest priority page(s) on your website and 1.0 identifies the highest priority page(s) on your website.
The URL check, server headers and other parts of the tool are covered on the Webmaster Tools page.
How To Create Google Sitemap
To create a Google Sitemap, simply enter the full address of your website, select any sitemap options and choose ‘Start Crawling”.
When you have completed the crawling process and status reports finished, then select export under the sitemaps tab and choose Sitemap XML.
Enter the filename you would like the sitemap saved and click save (the default is sitemaps.xml which is fine).
You have now created a Google Compliant Sitemap and saved it to your computer – congratulations!
Now, you need to upload the new sitemap to your website. It seems every hosting company has a different method of doing this, but they are all basically the same:
Think of your sitemap.xml file as any htm (html, php or asp) file that you’re going to place on your website.
There is probably some type of import option that your hosting company provides you – use it to move (FTP, Publish, etc) the sitemap.xml file from your computer onto your website. Place it in the same
directory that holds the main page for your website.
Submit your Sitemap to Google
The final step is activating your Sitemap. Here’s how:
- Log into your Google Sitemaps account by visiting Google Sitemaps Account.
- Click on the “Add a Sitemap” link.
- Enter the URL for your Sitemap in the field, then click the [Submit URL] button.
- Example: The URL for your Sitemap will be your website address, followed by the filename that you uploaded. For example, if I uploaded my ‘sitemap.xml’ file to my auditmypc.com site, the URL I would give to Google Sitemaps would be http://www.auditmypc.com/sitemap.xml
This will submit your Sitemap to the Google service. It may take Google a few hours to generate reports about your site, so be patient while they work their mojo.
How do I Resubmit my Sitemaps?
Initially, you just uploaded your site map file to your website and told Google the location of the site map. If you make a change, say add or delete information from the sitemap, you need to do the following:
- Generate another sitemap with the same name as the original
- Upload the sitemap file to your server just like before, but this time you will overwrite the original sitemap
- Tell Google you have made a change to the sitemap – you can do this one of two ways:
- A) Sign into Google Sitemaps with your Google account and from the “Sitemaps” tab, select the checkbox beside it and click the “Resubmit Selected” button.
- B) Send Google an HTTP request by typing in a special address into your browser (also know as ‘pinging google’)
To resubmit your Sitemap using option B (a HTTP request), you simply enter the full location of your sitemap that you provided Google and place it into your browser.
Encoded Sitemap
Once you have entered the encoded sitemap address into your browser and pressed enter (or Go, etc), you’ll receive a screen message from google telling you it has your information; Bookmark the address before you close it and next time you update your sitemap, just visit the bookmark. This saves you from having to go through this process again!
Encode your Google Sitemap
Although the form above encodes your sitemap url for you, you may be curious how exactly it works or want to manually encode your sitemap address.
As an example, if your Google Site Map is located at http://www.auditmypc.com/sitemap.htm, you’ll enter the following website address into your browser:
www.google.com/webmasters/sitemaps/ping?sitemap=http://www.auditmypc.com/sitemap.htm
but, google asks that you encode the URL first.
You must URL Encode everything after the /ping?sitemap=
So, you’ll actually enter:
www.google.com/webmasters/sitemaps/ping?sitemap=http%3A%2F%2Fwww.auditmypc.com%2Fsitemap.htm
Notice how the ‘/’ became ‘%3a’, that’s the hexadecimal representation of ‘:’. It’s easier to just use the form on this page to encode your sitemap location :)
In Summary, take the address of your sitemap, enter it into the ‘Site Map Location’ area above and press ‘Encode URL’ button. Append the encoded result to the end of google sitemap url like this:
http://www.google.com/webmasters/sitemaps/ping?sitemap=[encoded result]
To decode a Google Site Map address, simply enter the encoded url into the ‘Encoded Result’ and press Decode URL
For more detail on how to resubmit the Google Sitemap, simply visit Google Sitemap.
Yahoo Sitemap Generator
Yahoo also has a sitemap program that allows you to submit a list of your webpages to Yahoo!
Create a Yahoo Sitemap xml file using our webmaster tool.
Use Yahoo Sitemap XML files to make sure that all your web pages are indexed by Yahoo. This can be done by creating a single xml file containing your sitemap. XML Sitemaps for Yahoo is a free service and easy to do.
To create your XML Sitemap file follow these steps:
- Create the sitemap as normal using our Sitemap Generator (webmaster tool)
- Click the ‘Save Sitemap’ tab located under the ‘Sitemap’ tab
- Select ‘Sitemap XML’ to save it out as the name you would like and then upload the sitemap to your website
- Submit your XML File to https://siteexplorer.search.yahoo.com/submit
To use Yahoo Sitemaps, you will need a Yahoo account which is free. Then, add your website under ‘My Sites’. Once in ‘My Sites’, click manage and add your Yahoo XML sitemap file that you just created.
You can provide Yahoo Sitemaps with a feed in the many formats other than XML (stick with XML).
- RSS 0.9, RSS 1.0 or RSS 2.0, for example, CNN Top Stories
- Sitemaps, as documented on sitemaps.org
- Atom 0.3, Atom 1.0, for example, Yahoo! Search Blog
- A text file containing a list of URLs, each URL at the start of a new line. The filename of the URL list file must be urllist.txt; for a compressed file the name must be urllist.txt.gz.
Yahoo Sitemaps is easy and a great way to get indexed on this search engine!
Top Sitemap Builder Suggestions
Suggestions (Thank You!) from users that have been using the Sitemap Generator and that have been incorporated into the application include:
- Make XML Sitemaps for Yahoo Sitemaps, MSN and others.
- Add a Google Sitemap URL Encoder / Decoder to help submit sitemaps.
- Make the URL list sortable by clicking the column title. When generating a site map for larger sites, it can become overwhelming.
- The ability to save your settings when you create a sitemap. The next time you generate a sitemap, you’ll be able to refer to these settings.
- Include in the sitemap program the ability to honor robots.txt files. When a site map is created, the program will look at its own filters first and then look at the website’s robots.txt file if the option is selected.
- Include in the sitemap builder the ability to honor meta tag robot rules. When a site map is generated, the program will look at its own filters first and then look at the website’s robots.txt file and meta tag if the options are selected.
I want to thank everyone who has taken the time to suggest improvements!
Version history is included below.
Sitemap Bugs and Fixes
The current version is v1.4 released on 7/20/2010. Major items fixed include updated Google Sitemap code, regular expressions and load issues on initial start.
We provide free support for our sitemap generator tool, simply leave a comment and I’ll do my best to respond (I do this for free and time is limited, so no promises)
2010 – The Webmaster Tool now expands and contracts with the size of the page. Regular expressions are now possible with the use of the ‘complex:’ prefix. There was also an issue with the sitemap generator not loading on some browsers and is now fixed.
2009 – Sitemap Generator has been renamed to the AuditMyPC.com Webmaster Tool and is now at version 1.0. This new tool replaces the older sitemap generator and includes the ability to check sites for errors, server headers, XML sitemaps, and much more. If you select stay informed from the menu to your left, you’ll be able to use this tool on your site. Those not on the list will still be able to use the old tool.
2008 – Latest Sitemap Generator update is now at version v2.32. Fixed a case sensitive issue that could cause some internal pages not to be recognized. Fixed a ‘time out’ problem where the site map generator would not stop or just hang.
May 19th, 2006 – Version 2.3 – Performance issues such as speed and calculations made.
May 18th, 2006 – Fixed a ‘case’ issue with the sitemap generator that would see urls with mixed case as different from urls with straight case. For example, if you entered AuditMyPC.com and the site has links pointing to auditmypc.com/firewall-test.asp, it would be considered and external link and excluded from the sitemap builder.
March 28th, 2006 – Version 1.7 released;
- This version gives you the ability to look for and respect the rel=’nofollow’ tag. This tag is used to tell robots not to follow the link. The sitemap generator will now look for this tag and if found, will not include it when building a sitemap.
- In rare configurations of the robots.txt file, the sitemap generator would not catch some entries.
- Speed – enhancements to the speed at which generating sitemaps has been made and will also play a big part in the next major sitemap update.
December 1st, 2005 – Version 1.6 updated; when reading the robots.txt file, upper and lower case could cause a problem when building a sitemap. The sitemap generator is now case insensitive. Problem fixed.
October 28th, 2005 – Version 1.6 released; corrected a small bug that looks at a robots.txt file when generating a sitemap and if it sees ‘disallow:’, treats the command as disallow all. Although many people that generated sitemaps did not have this type of configuration, it forced the user to uncheck the ‘Follow robots.txt rule’.
September 23rd, 2005 – Version 1.5 released; this addressed speed issues when building a sitemap.
September 14th, 2005 – Version 1.4 released; no bugs, site map cosmetic enhancements.
September 8th, 2005 – Fixed a bug where the sitemap generator would hang if spaces were found in the url.
July 27th, 2005 – Version 1.3 of Google Sitemap
- Version 1.3 of the site map generator sports an entirely new skin, is more compact, has improved code and fixes a number of small bugs.
- One of the biggest requests by users was greater detail when errors were discovered. If your website has any errors you will now have the ability to see exactly what caused the error when attempting to build a site map.
- The number of invalid pages is easily identifiable. Each sitemap error can be expanded to see all related pages that point to that particular problem. You also have the ability to cut and paste Google Sitemap errors.
July 24th, 2005 – Updated site map submission instructions.
June 16th, 2005 – Identified and Fixed:
- Drop downs appeared to be missing from the sitemap creator when the user changed format from ‘Plain’ and then switched back to ‘XML’ format.
- Sometimes crawling startup failed when trying to generate a Google Sitemap.
- When creating a Google sitemap from a website that contains more than 4,000 pages, the applet started to consume more system resources. The sitemap creator now supports up to 50,000 pages!
- Added the ability to resize the site map applet to fit different size monitors.
June 15th, 2005 – Identified and fixed bug that when crawling over 3000 URLs, the Sitemap Generator would eventually drop down to one thread and may stop creating a sitemap entirely.
Google Site Map Generator v2
This version was RETIRED on 1/24/07 and was a major revision which included new features such as improved performance, the ability to ignore invalid relative links, ability to load from anywhere, max level, delay between requests, and the ability to save errors!
One of the most important and requested features have been the ability to regulate the speed with which the Google Sitemap Generator crawls a site. It’s possible that some website hosting companies may view multiple requests in a short period of time as hostile; this could result in the sitemap not being generated correctly.
You now have the ability to place a value in seconds so that sitemap building requests are spaced out over time and more friendly.
We found a large number of sites have invalid links, such as../ rather than ./ which can create additional pages when combined with rewrites. Many of the users experiencing this problem thought it was our sitemap builder creating extra pages and was not aware it’s actually a coding error on their site. Those users now have the ability to ignore these errors when building their Google Site Map.
We recommend you run the program without checking the ‘Ignore invalid links’ option to catch any errors. If our Google sitemap creator sees the file, so will the bots – it’s a great way to make sure you’re not generating duplicate content!
Another popular request was the ability to restrict the sitemap creator from going beyond (x) levels deep. For example, building a sitemap with a max level of 3 would go down three directories deep and ignore the rest of the sub directories.
By far the most requested enhancement is the ability to save out errors that have been reported when you build a site map. Site Map errors can now be saved so that you can review them later or work on them as time permits.
To access the new version of our free site map generator, simply click on our webmaster tool icon at the top of this page and you can start building your XML sitemap in minutes.
Sitemap Generator Support
You can receive support for the Sitemap Builder by leaving a comment below. Listed below are some questions that may still be valid.
One of the major advantages of using this tool is that owners of websites find errors they never knew existed on their sites! These errors most often prevent sites from being indexed by Google, Yahoo MSN and others! Once fixed, site owners usually notice a major increase in search engine traffic!
It starts off with trying to make a sitemap for Google and discovering that the sitemap generator simply stops at the main page and doesn’t find other pages within the site.
It is this very problem that people often write to me about. Almost always, after reviewing their site, I discover web pages that are missing beginning or ending tags, such as html, head and body tags.
I also discover that a large number of site owners are accidentally blocking robots from visiting their page. In the sitemap builder, you’ll notice that there is an option to honor robots.txt files and no follow tags.
If you’re having a problem with the sitemap builder and your page is formatted correctly, try deselecting the robots and no follow options. If this works, then the problem is with one of these items.
Every attempt has been made to make our webmaster tool behave as the search engine robots do when spidering a site. There are standards that each search engine subscribes to when reading websites and we subscribe to that same data. The point is, if we catch the errors, it’s a very real possibility that the search engines will also.
A perfect example would be a hosting company blocking our spider because it’s going to fast – chances are, the hosting provider is also doing this to the search engine robots and could be preventing them from seeing your entire site (which could lead to poor ratings). – See changing user agent above for solutions to this problem.
Creating a Sitemap – Common Questions
Question: I have created a sitemap and it includes files that I want hidden from Yahoo and Google. If I take them out of the sitemap, then the search engines won’t see them, correct?
Answer: Wrong – If the sitemap builder finds these pages that you want hidden, so will Yahoo, Google and other search engines. Sure, you can exclude them from the sitemap, but the problem is that you have links on your site to these files you want hidden.
You need to exclude them in your robots.txt file. My sitemap builder will respect the robots.txt file (obey it), just like the search engines and prevent them from being included in the site map.
Common Sitemap Builder Problems
Problem: You enter your website address and nothing happens.
Solution: Is that really your website’s main page? For example, you might have entered http://[yoursite.com] as the address, but if you type this in the browser and you end up at http://www.mysite.com/main/index.shtm, then you have a landing page that is different than your website address.
In this case, you would enter http://[yoursite.com]/main/index.shtm as the site address.
Problem: I want to exclude images and css files.
Solution: enter the following in the exclude area:
*.jpg
*.bmp
*.gif
*.css
Problem: You want to capture only urls that in the men sub-directory containing only numbers, letters and a forward slash:
http://[yoursite.com]/shopping/men/casual/21/2
http://[yoursite.com]/shopping/men/sports/soccer
but not:
http://[yoursite.com]/shopping/men/sports-2 (has a dash)
Solution: Use a regex expression by prefixing it with ‘complex:’, for example:
complex:http://[yoursite.com]/shopping/men/[A-Za-z0-9/]*$About regex:
- Entire url (including protocol and host) matched against pattern, for example:
complex:http://[yoursite.com]/shopping/men/[A-Za-z0-9/]*$- Pages filtered out are not processed, i.e. lets say we have root page that references page A, which, in turn, references page B, and page A doesn’t match filter rules, then you’ll will never reach page B.
- ** Quick Reference **
[A-Za-z0-9] = Alphanumeric characters
[A-Za-z0-9_] = Alphanumeric characters plus “_”
[^A-Za-z0-9_] = Non-word characters
[A-Za-z] = Alphabetic characters
[ \t] = Space and tab
[\x00-\x1F\x7F] = Control characters
[0-9] = Digits
[^0-9] = Non-digits
[\x21-\x7E] = Visible characters
[a-z] = Lowercase letters
[\x20-\x7E] = Visible characters and spaces
[-!"#$%&'()*+,./:;<=>?@[\\\]^_`{|}~] = Punctuation characters
[ \t\r\n\v\f] = Whitespace characters
[^ \t\r\n\v\f] = Non-whitespace characters
[A-Z] = Uppercase letters
[A-Fa-f0-9] = Hexadecimal digits- + Match one or more of the previous items (previous character) so, the expression Rob+in would return Robin, Robbin, and Robbbbin. Alternatively, you can build a list of Previous Items by using square brackets. Like this: [abc]+ This will return a, ab, cab, c, b, bbbb, etc.
- The carat (^) matches the beginning of the document.
- The dollar sign ($) matches the end of the document.
- A backslash (\) followed by any special character matches the literal character itself, that is, the backslash escapes the special character.
- The # and – characters must be escaped in expressions (## –) just as though they were special characters.
- A period (.) matches any character, including a new line.
- A asterisk (*) matches 0 or more of the preceding character (note that it will not be able to match an ending forward slash but period will).
Problem: Site Map Generation slows down after 20,000 pages
Solution: Some webmasters have noticed that during a crawl of a very large site, the sitemap generator may slow down after spidering about 14,000 urls. This can happen if the site is heavily nested or has a complex linking structure.
People who experience this lag usually have rapid applet memory consumption and need to increase the amount. The site map builder by default is limited to 50-100m which can quickly be consumed on a complex web site.
To solve this problem, you can increase the amount of memory used by the site map builder. Simply navigate to the control panel and click on the Java Icon. Then, inside the Java Control Panel, click on the Java Tab, Java Applet Runtime Settings, View and then in the Java Runtime Parameters cell, enter ‘-Xmx256m’.
You can take it a step further when building a sitemap (if you’re still having problems) and enter ‘-Xmx512m’.
Problem: You enter your website address and the site map builder stops immediately.
Solution: This is caused because your main page is redirected to another page (landing page). For example, you may have yoursite.com being redirected to yoursite.com/sales/products/sindex.htm
If this happens to you, simply enter your website address into your browser and notice where you are redirected to; take that redirected website address and enter it into the sitemap generator.
In the example I used above, you would enter:
yoursite.com/sales/products/sindex.htm
into the sitemap generator.
Problem: The sitemap generator misses a few or many files.
Solution: If you are having problems building a sitemap, it may be due to your Robots.txt file or your Metatag. Try unchecking the Follow robots.txt rules and/ or meta name robots rules.
Problem: I can’t see the webmaster tool graphic button, so I can’t start the test.
Solution: If that’s the case, then your browser settings may be preventing sitemap generation.
In IE, look under Tools, Internet Options, Security, Custom Level, Scripting of Java applets and choose prompt. Active scripting should be enabled as well.
In Firefox, look under tools, options, web features and make sure the Enable Java and JavaScript is selected.
If after trying these you still have a problem, please let me know and I will do my best to get you up and running.
Problem: Google Sitemap Invalid Date Error Message
Solution: If you get an ‘invalid date’ when you submit your sitemap, check to make sure that the time in not in the future. A common mistake is to not to account for daylight savings when creating the sitemap, so make sure you use the time zone for your server and not the local timezone.
Note: This sitemap generator runs on your PC and not the server.
Like our Google Sitemap Generator?
Like our sitemap builder? Please let others know by displaying this icon. Simply copy and paste the code snippet below onto your webpage:
Or, for the XML Generator
XML Sitemaps – 1kb at 80 x 15 in .gif format.
Feeling real nice? Then buy me a cup of Chai :)
Heh, just noticed that last comments comes on top. About my previous problem I think , if I change robots txt, I have to restart firefox in order that sitemap generator takes in new robots txt contence.
when you check respect: <a rel = "no follow"
does it mean that it respect "nofollow" without space, a sit should ?
It would be good to have additional :
check respect: <a rel = "noindex"
regards
Hey MC,
I can’t help you with a web address to look at – I need to see your setup in order to solve your xml sitemap problem.
Let me know…
my robots.txt:
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /img/
Unless I check off “respect robots txt” sitemap generator crawls nothing ?
You are welcome Andy!
Grazie per la sitemap
thank you
hi
do you know you have done so much good to all novice and fresh website owners and general public by posting all this information at one place. Truly knowledge like happiness is doubled when shared. Thanks a ton buddy.
hats off to you and your effort
Hi Chris,
Yes, looking at late next month if time permits…
Thanks!
Have you got any plans to include video sitemaps into this great tool?
Hi Mark,
You exported a “save project file” rather than the XML Sitemap Google needs. Here is what you needed to do:
1) Run the sitemap generator on your site.
2) Click on the Sitemap Tab
3) Click on the Export Tab
4) Select Sitemap XML from the drop down list
Those are the steps needs to create the sitemap xml file that you’ll submit to Google.
Hey Mark,
So what exactly is the name of the sitemap file? Or better yet, what is the exact website address of the sitemap file and I’ll take a look.
I ran the sitemap generator and cleaned up problem area. I set the change frequency on the pages that change frequently (weekly). I then downloaded the sitemap to my site video-games4u I submitted it to Google through the Webmaster Tools. Google gave me the following error.
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.
My robots.txt file shows the following.
User-agent: *
Allow: /
I don’t understand what is causing the problem. I looked at the sitemap.xml file and could not see anything wrong.
Regards,
Mark …
Hi Asha,
When you visit the page with your browser, you’re making one request, but the sitemap generator default to 5 requests at the same time (can go up to 9 requests), so your results doing it individually will be different. What you can do is bump up your Request Delay from 0.0s to something like 1s – Look under the Crawler Tab for these settings.
Thanks for sharing and the kudos on the software :)
I had problems with low memory also. I upped the max jre memory usage to 256 meg (java -Xmx256m) which solved the problem.
The issue I cannot solve is the generator tends to give me many timeout errors waiting to connect when crawling my pages, even though when I go to the pages directly via web browser the page is displayed without issue. Sometimes I have to reload the sitemap generator a few times before it works reliably. Other than that it’s a great piece of software. Many thanks for releasing it.
Hi Jim
Have found your sitemap generator because I wish to submit a sitemap to Google.
My site is worldaudiobookclub .com which has 12000-14000 audiobook title, which is not the problem. As each page open – with each unique title as its centrepiece – a window of New releases, Recommended and Bestsellers also appear to accompany the selected title. Each new visit to each pages brings a refreshed selection, meaning that each title can appear once as its own title ID, but many other times in the adjacent panels, randomly populated by ‘the system’.
Thus, my sitemap recognizes approx 17000 URLs, but each of which can have up to 20+ extra titles.
This may well be creating a maze more than a map!
Your software appears to queue them all correctly, but after about 8000 ‘finished’ results, it stalls and won’t go further. The last 50-80 entries list as ‘Failed’ – Low memory…
Is this because my PC’s memory is full, there is a limit om the size of the XML file that can be created, or what? Even better, what settings do I apply to not include the extra links, or conversely, to only include the titleID pages?
Looking forward to your help. And I’ll get you a Chai as well – it’s one of my favourites.
Kind regards
Stephen Barrett
New Zealand
Thanks Jim i have been looking for months for such a tool and i can’t believe the price. I know time is a very illusive creature but let me say your service to the community as a whole is priceless! I have been struggling for a while now trying to get my site indexed by the big 3 i’m going to use this tool regularly, i am sure i’ll have some questions before i am satisfied with results but i cannot stress how much i appreciate your service and information you provide.
I’ll proudly put a link in the footer of my site for you!
Thanks, Thanks, Thanks
Hello..Good service bu i have some trouble.
On my site have russian words and sitemaps xml notunderstand my language.
So what i can do?
My sitemap.xml file does not display as a formatted xml file in the browser. Am I missing something? It displays as just one long string of text (VERY long!).
I can’t get what to do. I mean how to begin that process. I don’t see any link where I can access the interface of sitemapgen.
Hi Bruce,
At this time, only test/html files are included in the xml sitemap. The PDF is a application/pdf type.
It is on my list to add this down the road, but time is an elusive friend.
Hi Jim
RE- XML Sitemap Tool
Although I successfully created a sitemap.xml file for my web site using your Sitemap Generator, when I saved the sitemap file, NO listings were included for the many PDF files that I have on the web site.
However, all [paths to] PDF files on the site were included in the generated sitemap list… just not saved to the sitemap.xml file.
How to I force the generator to include PDF files in the final sitemap.xml saved to my hard drive?
When I used the limited edition Google Sitemap generator, PDF files were included in the sitemap.xml file.
Thanks
Bruce
Great site map tool and a real treasure for an enthusiastic but amateur webmaster.
Although I successfully created a sitemap.xml file for my web site, when I saved the sitemap file, NO listings for the many PDF files I have on the web site were included..
However, all paths to PDF files on the site were included in the generated sitemap list… just not saved to the sitemap.xml file.
How to I force the generator to include PDF files in the final sitemap.xml saved to my hard drive.
Thanks
Bruce
I like your tool and use it when creating almost all of my sitemaps. I have a website I’m using it on which keeps returning not only the http versions of the urls, but also the https versions of the same urls. I have put code in most of the files it’s returning to generate a dynamic META robots tag with “noindex,nofollow” when accessed via port 443. This seems to work fine via the browser (I see the tag in the source), but the sitemap tool still lists these urls even though I have the box checked to use the meta robots tag. I’m trying to figure out if the search engines would be doing the same thing or if this is an issue with the tool, or if there’s more I need to do with the code.
Hello
I have used XML Sitemap Tool. Could you tell us more about this error :
“unexpected end of file from server”
Regards
Remi Brandini
I have a problem with my website. When I burn my rss feeds to feedburner.google.com I always encountered the error: The URL does not appear to reference a valid XML file. We encountered the following problem: Error on line 15: Open quote is expected for attribute “{1}” associated with an element type “language”.
I have checked already my template code and I have even deleted already the meta with language in it but still the problem.
Please help me.
Nice sitemap builder, Thanks!
Hi. Nice generator. Does it create multiple sitemap files on export? I heard Google has a 50k limit per file so we should use multiple files.
It is very nice website for making site maps and found it easy to use, thank you.
Re: XML Sitemap Tool (Cont’d)
Hi Jim,
Using Firefox, set to level four, your app continues to level 5 (at which point I stopped — level five takes many hours to complete, even with 9 threads).
Thanks again for your help and support.
Basil
Hi Jim,
When you choose export, a save window pops up and will save it in your default documents folder as New sitemap_sitemap.xml IF you don’t choose where you want it stored. To choose where you want it stored, just click the yellow folder with the green up arrow in Windows (standard icon and navigation) to choose a different folder. To choose a different filename, simply change the name (it is highlighted by default).
Whenever I export my sitemap i can not find it so i can upload it to my website. I have tried saving it to my desktop and to my website directory on my PC and still can not locate the file.
Thanks for the comment Basil!
I fixed the screen in IE – thanks for pointing that out to me!
As for Firefox, what happens when you set it to Max Level = 4?
Best,
Jim
Bug Report XML Sitemap Tool
===========================
I do find that your sitemap tool is the very best — thanks for your effort.
A couple of bugs in the new version:
Running in Firefox v3.6.8, Allow *.html, *.php, disallow *.js *.css no images, starting at the root, if I set Max Level = 5, it continues beyond to level six… hundreds of thousands of entries being generated. With php and a large database and many combinations and permutations,, max level is important to keep the number of pages indexed to about 50K, the more basic entries.
Using IE v8.0.6001.18702, (same settings) each tabbed window shows up as perhaps several thousand pixels high. Your app. has its own separate window in IE (whereas in Firefox, it uses the main browser window). In IE, your app’s scroll-thumb can be off the bottom of the IE window — perhaps challenging for the neophyte as some of the option settings aren’t visible. Max Level seems to work under IE.
(I haven’t tried your app using Chrome or Opera).
I’m using XP Pro, SP3, Java build 1.6.0_21, memory set to -Xmx1024m, Intel dual core E6750 cpu, 4GB RAM.
Thanks again for a fine program!
Basil
Hello,
I just wanted to thank you for providing google Sitemap Generator.
And I thank you Jim for your detailed and very helpful answer. I was going to post the same query as Jerry, I faced a similar problem.
Hi, Thank you very much. I was at a loss as how to generate a sitemap. This has worked in Google so I now have to submit to Yahoo etc.Regards, Pauline
I tried it and this is what Google responded with Unsupported file format?
I often use an service internet to generate an xml website. This is a great tool , thanks you very much.
I am thinking of create an php CRON JoB that can generate sitemap on the server … Can you help me ?
Hello,
Quick Answer: No
Long Answer: Anyone can download your sitemap file if they know the name of it and now people are referring to it in their robots.txt file, which anyone can see. If the sitemap is not in the robots.txt file *and* named something other than sitemap.xml *and* that name is not referenced anywhere on your website then only the search engines you registered the sitemap with would know the name.
Keep in mind, that passwords are not included in sitemaps, so even if someone did get your sitemap file, they could not figure out your passwords.
Hey I was just curious, if you have a sitemap for spiders to follow where everything is on your site, is there a way that someone could download that file then be able to access a site that had a password to enter it?? Just curious….
Hello Jerry,
You exported a HTML file or something other than the sitemap xml file that Google needs. Here is what you do:
1) Run the sitemap generator on your site.
2) Click on the Sitemap Tab
3) Click on the Export Tab
4) Select Sitemap XML from the drop down list
Those are the steps needs to create the sitemap xml file that you’ll submit to Google.
I tried it and this is what Google responded with:
Unsupported file format
Your Sitemap does not appear to be in a supported format. Please ensure it meets our Sitemap guidelines and resubmit.
FYI: You have a bad link under the “Submit your sitemap to Google” section.
Log into your Google Sitemaps account by visiting Google Sitemaps Account.
The link should be
https://www.google.com/webmasters/tools/home?hl=en
Thanks for the great tool!
Hi Jos,
I’ve added the regular expressions commands to the main page above, so check those out under include filters. I have some examples that will help you.
Thanks,
Jim
Hey i am trying to generate sitemap for a website with 50k+ pages, so i want to break it up by subdomains. But not sure how to generate a sitemap for a perticular subdomain only.
Thanks
Regards
Nik
Hi Ted,
You have to start a new project – if you simply run the sitemap tool again, it won’t change any errors, just display new ones. Just start a new project and you’ll be all set.
And yes, the ability to find your errors and what has caused them is my favorite (along with titles and times).
Thanks,
Jim
Thanks Ted, I’ve made the change!
The Yahoo submission link you have as part of the instructions above is no longer available
I found it at – siteexplorer.search.yahoo.com/submit
Take care,
Ted
First, thank you! The video was very helpful in explaining your very useful tool.
Being a professional software developer and loving to get feedback from my users, I have two things I wanted to let you know about:
1) When I went to the “Sitemap” tab, I had to scroll the window down to see the “Buttons” and “Report information” at the bottom. This behavior was the same in Google Chrome v5.0.375.99 and Internet Explorer v8.0.7600.16385 . Both were run in a maximized window.
2) As in your video, my sitemap generation showed me an error (Wow – I absolutely love that feature). I fixed the problem on the website and wanted to re-run the sitemap generation. I thought “Clear entire sitemap” (Right click on the sitemap results) might get the software back to a start-state, but it did not appear to. It showed the same problem – even though the problem was fixed.
Thanks again for a great utility!
Hi Emran,
You need to prefix your regular expressions with “complex:”.
Try this command:
complex:http://[yousite]/shopping/men/[A-Za-z0-9/]*$
If you don’t specify it with complex: then whatever you enter will be treated as a simple expression.
Let me know how that works for you :)
Thanks,
Jim
Hi Jim…
Trust all is well, just wondering if a fix for the regular expression issue had been posted?
Hear from you soon.
Regards, Emran
I tried using your sitemap generator for [snip] (please keep private), but I had a problem similar to others who posted comments earlier. It just says that I sessions are respected and I should log out before proceeding, but it gives no way to proceed.
Thanks.
Jim – thats fantastic !!!!
Hi Emran,
Yes, I’m looking into this and will post a fix – thank you for your descriptive comment, it makes it a lot easier to test!
Hi
I’m trying to generate feed for my website, however I only require the product pages for a specific top level category.
I’m using the include filter option to enter the following regular expression:
/shopping/men/[a-z]*/([0-9][0-9]|[0-9])/([0-9][0-9]|[0-9])
this should return the following examples but doesn’t.
http:// [mysite.com]/shopping/men/casual/21/2
http:// [mysite.com]/shopping/men/sports/1/21 and so forth
The regular expression has been checked against an online validator and works fine.
Any ideas? Please help
Jim, it means if i do it well, it is correct or not, Should i add my xml URL into google.
Tnx
Hi Jim, just to let you know I did have to reinstall Java, works fine now. Thanks for everything!
Hi Robin,
You need Sun’s Java to run the tool.
Visit http://www.java.com/en/download/installed.jsp and let me know what version you are running.
Thanks,
Jim
Hi Jim, I don’t have java.exe running, its still just giving the sentence on top of browser, nothing else.
I tried in IE as well.
Hi Robin,
This issue has been fixed and you should not encounter it again.
This happens to me from time to time and I have no idea why, but my solution is to kill the java.exe process (I use http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx ) and close all my browsers, then try again. There have been times when I needed to do this multiple times before it finally takes.This started happening with the new version of Java a while back and I’m still investigating why and how I can fix this.
Do me a favor will you? Let me know if this works for you and how many times you have to do it.
Best regards,
Jim
Hi Jim, sorry to bother you at this email, but I was going to use your xml sitemap generator. I’ve used it in the past because it seems to index stuff my version of xml sitemaps doesnt such as 301 redirects so I hear.
Problem is In both Firefox and IE im getting this only on the page
This tool respects sessions, so make sure you are LOGGED OUT of your website BEFORE generating a XML SITEMAP!
A blank page comes up when I click on the icon, only the initial message at the top is showing.
Any ideas would be appreciated. The applet isn’t showing up.
Kindest regards
Robin Downey
When you say placed well, what exactly are you looking for Suffi?
Hi,
I’m not the webmaster, But i have generate the XML site map for my site and i still don’t know if everything’s placed Well or Not, Can anyone help me
my site: irtouring. com/textpage.asp?pageID=36
Thanks
Good Stuff!! Helps me improve more sitemap generation!
Got it – I’ll work on this later and let you know what I came up with. It probably won’t be until after the 4th…
Enjoy!
Currently I have a sitemap containing every single html page, about 4000, which I want to reduce to only the introduction pages, starting a subject in three languages and the index pages of all photo albums, but no longer every html page that contains the enlarged photo.
Regards, Jos
I’m sorry Jos, it’s Saturday, have family and friends coming over soon and not into the problem solving mode like I usually am. What exactly do you want to do? Have a sitemap with only index files? I’m trying to get this wrapped up before everyone arrives.
Hi Jim,
No, it’s not.
I entered the url vanderburgt. eu/holidays/citytrips.html
Then I only use include filter *index* and put a checkmark in the box exclude images. It gives me 17 html pages that lead to the index thumbnail pages mentioned on the page.
Removing *index* gives me all html pages on the website.
Regards,
Jos
Hi Jos,
Is the url you are working on the one you mentioned in your previous comment? If so, I’ll try and create a xml sitemap for that – also, send me the excludes / includes you are using.
Thanks,
Jim
Hi Jim,
If I don’t use any filter all urls on the entire website are found, but as soon as I put *index* in the filter box only “index” pages will be found on the page I entered in the URL address box. No links are followed.
Regards,
Jos
Hi Jim,
Thanks for looking at this.
OK so I ran it on your testingiam.com site and it worked perfectly, and reloaded the saved project file with no problems
I ran it on my site, but added a few more excludes so that I only had 3,000′ish pages, it worked perfectly and reloaded the save project file with no problem.
I ran it again on my site and reduced the number of excludes giving me 13000+ pages in the sitemap, and it also worked perfectly and reloaded the project with no errors.
If you changed something then it has fixed the problem, if not, the problem must have been at my end and I apologise for wasting your time. I have no idea what the problem could have been unfortunately as I don’t believe that I changed anything between the two attempts at indexing the site.
All I can think of is that I may have had apache and mysql running when the reload failed, so my system may have been a little constrained on memory.
Thanks again for the tool, and sorry to waste your time.
Regards Chris
Hi Jim,
The .htaccess file was something I wondered about myself in a message to you. I’m afraid it got mixed up on this page containing different posts about various subjects.
The only error I get is testingiam.com/../index4.html but testingiam.com/index4.html was found.
Regards, Jos
Hi Jos,
Did I mention the .htaccess file? I reread my response to you and didn’t see anything like that…
And when you ran the sitemap generator on testingiam.com, there should be three errors which are included on purpose, they are index4.html, tesing.html and product1.php
When you run the sitemap generator on testingiam you should receive the same results.
the .htaccess file cannot be the problem, because I removed it temporarily while trying to generate a sitemap. Still no results using filter *index*.html
Thanks – Jos
Hi Chris,
I put it to the test, added my filters, ran the sitemap and saved out the project.
I then selected New Project, then loaded my saved project file (New sitemap.xml) and evertyhing was there.
How big is your project? The site I gave you testingiam.com, is a small site I set up with valid and invalid urls for testing purposes. If you run the sitemap generator on testingiam.com, do you also receive an error?
Thanks – Jim
Hi Bjantiques,
This can happen from time to time and is often a result of an old version of Sun’s java. If you have the latest, then you can terminate the java.exe process and try again. I should mention, that I’ve had this happen (on occasion) even when the latest version is installed – I simply terminate the java process and try it again. Why this happens I have not a clue, and apologize.
Addendum info found after further investigation.
This may help cure the issue at hand.
I opened page speed and found this error message
Avoid bad requests
The following requests are returning 404/410 responses. Either fix the broken links, or remove the references to the non-existent resources.
* auditmypc. com/jmaster.webtool.app.WebToolApplet.class
when on this sitemap generator page, I do as instructed and click on the image that has on it Start Now Click on the image.
This opens a new tab in IE then I see a java image ( i assume it is flash ) displayed and then – nothing more happens. If i try to close out the tab or the browser nothing happens apart from the usual MS dunk sound that tells you it is not going to react.
I have to use Task manager to terminate task.
With FireFox it opens a new tab shows the printed warning about respecting cookie sessions and then yet again – Nothing happens.
The good part is that I can close the Tab so its not locking up my FF browser.
Please do not ask me what website it is failing on as you did the others with the exact same problem as we have not got that far – before putting in a website you have to get the initial page open.
Thanks for the reminder Chris, I completely forgot this and will do so today. If you don’t see something here tonight, then bump me – I’m working on converting a big site and it’s taking more of my time than I ever expected!
Hi Jim,
RE: java.lang.NullPointerException on project reload
Did you manage to duplicate the problem?
I am not hassling for a fix but if you cannot duplicate the problem then I should look at something locally to see if its a problem at my end.
Thanks
Hi Jim,
On the sitemap tab I found 8 html files and 1 “failed” entry on testingiam.com using *index*.html.
So there must be something something else wrong.
.htaccess perhaps?
Regards,
Jos
Hi Jos,
I set up a test server you can run the sitemap against. Check out testingiam.com and for the include filter, use:
*index*.html
You’ll notice that the files:
indexme.htm
index.php
where not included, along with anything that did not have index in them.
Once you run the sitemap on this site, let me know the results, then I’ll look at yours – I want to confirm you are getting the same results as me before we dive deep into this.
Regards,
Jim
Hi Jim,
An example of a index page containing tumbnails is
vanderburgt.eu/croatia/svetijure/index3.html
Jos
Hi Liddia,
Here is a copy of a robots.txt file with the sitemap included. When the bots, be it Google, Microsoft and any other search engine, visit the site, they look at the robots.txt file to find out what they should include. When they see the sitemap heading, they will read that and may use it to help spider your site.
How do I include the file in my robots.txt file?
Hi Jos,
Can you give me an example of your site and I’ll take a look – by the way, you do not want to use index anything if you can avoid it. I’ll explain more once I see your site.
Best regards,
Jim
Leave a comment if you have any questions Rajesh.
I am new to the world of webmasters and came to know about site maps in your site. i am going to create one for my site. thank you for the information.
Hi Jim,
Obviously I have a problem understanding wildcards.
When I use include filter *.html all html pages are included.
If i want to include only pages like index.html or index2.html or indexwhatever.html I thought *index*.html would be the right definition, but unfortunately nothing happens. What can I do?
Regards,
Jos
Hi
I set the row filter change frequency and the priority but it does not transfer to the site map I have to put these in manually, is there a way to have these set as a default.
Thanks
Peter
Hi Chris,
I’ll run a sitemap on one of larger sites and see if I can duplicate the problem. Thanks for taking the time to make me aware and for the kudos :)
Jim
Hi Colin,
Glad to hear it’s all working! and thanks for the mention – may good rankings come your way :)
Jim
thanks Jim,
it worked this time – I must have used the ‘save as’ option under ‘file’ in the browser rather than the Java ‘Export’ button.
Excellent tool!
I’ll do a blog entry for webmaster tools and link to your site for my readers.
Col
Hi,
I would like to report a few problems when reloading a project from disk.
I always get a java.lang.NullPointerException when reloading and it never replaces the project details page which contains the include and exclude lists.
I assume this is done after the null pointer error is received.
The sitemap page appears to be fully reloaded.
I have tried this in Firefox and IE8
The project in question contains 10,000+ pages so its not small but also not the largest on the web either.
I am running:
Firefox 3.6.3
IE 8 fully updated
Vista Ultimate SP2
1 gig mem
Java memory set to 384m was 256 but i upped it to see if that would help and it didnt.
website = danastock.de
Hope you can help because otherwise this sitemap tool rocks.
Chris
Hi Col,
Once the sitemap generator is finished indexing your site, choose Sitemap, Export then Sitemap XML and give it a name that you’ll remember. Now upload this to your website and tell Google or Microsoft where your XML file is located (that would be your website address/name of your exported file).
Click on the image to enlarge.
This works like a charm. Also, I just created a sitemap with different priority settings (1, .09, .08, …) and submitted it to Google and it was verified and read shortly after.
Hope that helps :)
Jim
Hi Mary,
I’ve thought about that but using a plug would not allow you to see your site like the search engines do, entirely… My sitemap generator is also a tool for website owners. It allows you to see response times, discover errors such as invalid relative links, html errors (including what caused those errors), titles / mimetypes, encoding, incoming and outgoing links and a host of other information – much of which would be lost if I made this into a plugin.
If all you need is a basic sitemap for WordPress, then a plugin is the way to go and faster; however, if you want your site error free, are working on optimization and want to see your site the way the search engines do, then my sitemap generator is the tool to use.
By the way, if you are generating a sitemap for a site built using WordPress, you’ll find these exclusions very handy!
*/trackback/*
*/feed/*
*/feed
*/comments/*
*/tag/*
*/author/*
*/wp-content/*
*xmlrpc*
*wp-admin*
*.jpg
*.ico
*.gif
*.jpeg
*.css
*.xml
*.zip
*.swf
I am using WordPress for my website, do you have a sitemap generator plugin?
just seen this on the Google sitemaps help for this error:-
“That the namespace in the header is “http://www.sitemaps.org/schemas/sitemap/0.9″. Note that this must end in 0.9. If it ends in .9, you’ll see an error.”
That namespace doesn’t seem to appear at all in my file??
Hi Col,
What is the name of your sitemap and I’ll take a look and tell you what’s going on.
If you don’t want the url posted for all to see, just let me know in the comment.
Best regards,
Jim
hi,
saved a XML Sitemap listing about 9000 pages at telecomsadvice.org. uk, put it in the index directory and submitted it to Google – got the same error message as Gerry above – unsupported file format.
When I opened the xml file to look at it, all the URLs and descriptions are gobbledegook (or is that encoding) unlike other xml generators I have used.
Any clues as to where I went wrong please
I added a new certificate to the sitemap, so those of you that were receiving an invalid certificate should no longer have this problem.
The problem was that without a valid certificate, visitor’s would receive a warning before running the sitemap generator. To remove that warning, it runs about $300 per year (not fair).
Certificates cost money, and paying $300 per year was not worth having that message removed but recently, I found a code signing certificate cheap so all should be good for the next three years. Enjoy!
doesn work in firefox 3.6.3 got it to work in IE
Nice info provided my friend. going to create my sitemaps following your info. thanks for posting
sunbizar-technologies. com, how do I add sitemap on this site?
Hi Betsy,
I can’t help if I can’t see the website you are trying to build a sitemap for. Leave me a comment and I’ll take a look. If you want the website address to remain hidden, then just say so in the comment and I won’t publish it – everything here is moderated, so it won’t show unless I approve it – too much spam…
Jim,
Unfortunately, I’m having the same problem as Brian.
When I press the”sitemap generator tool” button, I get a blank page with the heading, “This tool respects sessions, so make sure you are LOGGED OUT of your website BEFORE generating a XML SITEMAP!” Nothing else on the page — no buttons/links/or sitemap tool.
I do not use log-ins with my website, I have the required java add-on, I use firefox as a browser, I’ve cleared my cache & data. What should I do to get beyond this dead-end stop? I would love to use your tool :)
Hi Brain,
There should be no problem with multiple IP addresses and the warning is simply letting you know that if you are logged into the site, something that has an administration section. Some content management systems, when logged in, will present a delete page or delete post button without confirming your request (sign of poor coding); in cases like this, the sitemap generator may follow the delete link and you would lose your data. So, I issue a warning that it respects sessions.
So, I’m guessing you are having a problem and associating it with this, but that’s probably not the case. If you leave me a comment with the site(s) you are trying to generate a sitemap for, I can take a look.
Better yet, can you word your question another way, perhaps I am not following :)
Hi there I host my severs from my home and I have 5 static ip addresses and when I go to create sitemap I get the error:
“This tool respects sessions, so make sure you are LOGGED OUT of your website BEFORE generating a XML SITEMAP!” anyways around this? Als I made sure I was logged out and on a different ip address. Is there any way to solve this?
Jim, Thanks for the tip, I had no idea!
:) Jill
Hi Jill,
Easy Solution! Simply add this to the include filter:
*/~irlmayo2/*
Whenever you have a website hosted off of a main site like rootsweb or sites.google.com/site/[Your website] and you want to generate a sitemap, you need to use filters – in this case, I used the include filter.
This include filter tells the sitemap generator to retrieve anything that starts in the /~irlmayo2/ folder.
Enjoy!
I used your sitemap generator once before and it worked quickly and quite well..this time when I put in my websites URL…. it looked like it was trying to index the entire rootsweb genealogy website.
I put in my site rootsweb.ancestry.com/~irlmayo2/
Any ideas as to what went wrong??
Hi,
I uploaded my sitemap.xml to my ftp.
I then submitted the url, bnbliving. com/sitemap.xml, to Google (dashboard/submit sitemap).
I got the following error:
Unsupported file format
Your Sitemap does not appear to be in a supported format. Please ensure it meets our Sitemap guidelines and resubmit.
Any idea where I might have gone wrong?
Thanks,
Gerry
what is this
Thanks! This was fast, easy, and only missed one page…
I try to export the sitemap from the generator to a folder I created on desktop, but i cant find the xml in the folder… so how I can get the xml file from the generator?
Hi and tx for great tool you share with us!!
Still, I could not find enough information, on cases where the program won’t index big part of your site – even though the pages are not password related.
I deselect the robot, meta etc – of-course
Example: in helena4love.co.il there are about 5000 pages of the type (example) helena4love .co.il/user_page.asp?site_lan=&user_Id=12588
that the sitemap generator won’t index.
Any tips will be more than appreciated.
Gidi