My latest article at Ecommerce Developer, read it in full here.
The first thing I do when working with a new site is set my favorite crawler on it. This gets me acquainted with all the URLs, site sections, interlinkings, forgotten pockets, scars and warts. A good crawler offers a wealth of data useful not just to search engine optimization, but also to site maintenance in general.
Luckily, some great crawlers are free. You’ll find pages of options just by Googling “web crawler” or a similar term. Xenu Link Sleuth is my favorite for the price — it’s free — and for the broad assortment of data collected on every URL it crawls. GSite Crawler is another good, free alternative. It’s focused mainly on creating XML sitemaps and feeds, but it’s good for other uses as well.
Originally posted on Web PieRat.