Can information I posted years ago on a social site be cached and found on the deep web?

In this excerpt from
Answercast #21, I look at the idea of a “Deep Web” and what information you
might find about yourself there.

]]>
<![CDATA[

Deep Web

So there’s a number of interesting issues here. I too had not heard the phrase “Deep Web,” although I’ve certainly been familiar with several of the concepts.

Ultimately, here’s my take on it. If an individual can view your web page without going through any special hoops like say, connecting to a special network, or entering a username and password, or anything like that, then I’m convinced there’s a search engine that has already spidered it and potentially cached it.

There’s nothing to protect it from that kind of activity.

Limiting website spidering

What your admin may have been talking about is this thing called robots.txt. It’s a file that website owners can place in their website that tells search engines what can and cannot be spidered, or can and cannot be cached. There’s also information you can place on an HTML page that basically says the same thing. And that will keep the pages out of the popular search engines.

You will not find those pages in Google; you will not find those pages in Bing.

The problem is that techniques, like robots.txt, and the information that is placed in the HTML file is purely voluntary. It’s not a technological solution. It’s basically telling a search engine, “Hey, please don’t index me. Please don’t cache me.”

Good search engines will do that; will respect that. Others may not. There’s actually no requirement that they do so. It is simply a gentleman’s agreement.

By the fact that your website, or your content, is available without any special hoops to jump through, I’m certain, I’m convinced that there are search engines, caching utilities, spiders out there that have and will continue to spider your site and cache its content.

Programs crawl the web constantly

There’s really not a lot we can do about that.

We normally think of there only being a handful of search engines; you know, things like Google and Bing and maybe Yahoo back in the day. In reality, there are thousands and thousands of search engines and spiders crawling the web almost constantly.

Everything from:

• Competitive search engines to
• Government sponsored data collection utilities to
• University research projects that are out there just sort of surfing everything they can find.

So, in reality, the fundamental rule of the internet still applies.

Once you post something that is publicly accessible, for all practical purposes, you have lost control over it. You can remove it, but you do not know if somebody hasn’t already cached it, made a copy of it, or reposted it somewhere else. That’s just the nature of how the public internet works; dark or otherwise, that’s just how it works.