Horror, Spiders, and Robots

robotic spiders
Dave Wright - Web ManagerWritten by: Dave WrightWeb Manager

Ready for a scary story, with robots and spiders? Not exactly science fiction, we'll call this one "web development reality" instead.

Search engine companies like Google have developed computer programs that go online, and see what's out there on the internet. These programs check every page, click every link, gather as much data as possible - and report back to headquarters with the information. These programs are called "robots," and they're sometimes referred to as "spiders" because they deal with the web.

Why should you care? Well, because spiders can hear you, even the robotic variety! You can say, "Hey, when you're looking at my website, stay out of the folder called 'Private' please."

Or, maybe you're working on a new site, and you don't want your not-quite-done website to come up when someone does a search for "cool new tiki bar in Phoenix."   In that case, you could tell the spiders to stay away. Then, on opening day (when you cut the ribbon and start pouring drinks) you can tell the spiders it's fine to start collecting data on your site and adding it to their search listings.

And now, for the frightful tale: One dark and stormy night last year, a client came to us asking for advice - specifically about search engine optimization (SEO). It was then that we discovered the horror - their site, which had been up for more than 5 years, had been telling the robots, "Go away, you're not allowed to look at this site!" We took down the "Go away" sign right away, allowing the robots and spiders back in, and they started seeing noticeable improvements in their search engine ranking not long after that.

Why? It is hard to say because we didn't build the site originally, but my guess is the person or company who built it put up a "Go away" sign while working on the website, then simply forgot to take it down when the site was launched. Also, there was no final launch checklist used to go back and check for that sort of thing. Maybe the client's uncle's friend had a coworker who did websites, but the attention to detail was just not quite there.

Unfortunately, the consequence was missed opportunity - just imagine how much difference 5 years can make. They could have been ranking higher, been getting more calls, and making more money.

The takeaway? Robotic spiders are your friends - you just have to make sure that what you're telling them makes sense. If you want to be found online, don't tell the robots to go away. And, it's best to work with a marketing agency that does it right, so you don't run into any scary scenarios. Ready to get a quote?