|
by Nathan Chandler
2015
from
HowStuffWorks Website
Spanish version
|
Nathan Chandler is
a freelance writer and photographer based in Lincoln,
Neb. He earned his B.A. from the University of Northern
Iowa, and has researched and written about consumer tech
for more than 10 years. |
The "Deep Web" is Not All Dark
The Deep Web and the Dark
Web are topics of intrigue, particularly since the FBI
shut down the Silk Road website in October of 2013.
Many inaccurately use the
terms Deep Web and Dark Web interchangeably.
However, the Dark Web is
only a small portion of the Deep Web (below
Figure).
While both the enormous Deep
Web and the much smaller Dark Web are inaccessible
to Surface Web crawlers such as Google and Bing, there
is a distinct difference between the Deep Web and the
Dark Web as these contain different types of information
and different ways to access this information.
Nathan Chandler, in the
below article, on how the Deep Web works, contrasts the
Dark Web with the Deep Web:
Layers of
the Web
"There's a flip side of the
Deep Web that's a lot murkier - and, sometimes, darker -
which is why it's also known as the dark Web. In the
dark Web, users really do intentionally bury data.
Often, these parts of the
Web are accessible only if you use special browser
software that helps to peel away the onion-like layers
of the dark Web.
This software maintains
the privacy of both the source and the destination of
data and the people who access it.
For political dissidents and
criminals alike, this kind of anonymity shows the
immense power of the dark Web, enabling transfers of
information, goods and services, legally or illegally,
to the chagrin of the powers-that-be all over the
world."
In light of the Epix
Documentary 'Deep Web,'
released on May 31, 2015, we want to emphasize that
Deep
Web Technologies, a company founded in 2002, with some
of the most prestigious companies in the world as
customers,
is not involved in nor has ever been
involved in any efforts to access the nefarious parts of
the Deep Web known as the Dark Web.
On November 5, 2013, in
response to a Time Magazine cover story entitled,
The Secret Web: Where Drugs, Porn
and Murder Hide Online, Abe Lederman,
Founder and CEO of Deep Web Technologies, wrote a
blog article and a letter to the Editor of
Time raising concern about the inappropriate use of
"Deep Web" to refer to the "Dark Web".
On May 26, 2015, in
anticipation of the Epix Deep Web Documentary, Abe
Lederman, wrote a another blog
article, once again pointing out the
confusing way that many refer to the "Dark Web" as the
"Deep Web".
Deep Web Technologies has
not had any interest, in developing capabilities to
access the Dark Web.
If you want to learn more about
Deep Web Technologies, the company, and not the Epix
Documentary try this
Google Search and review some of the
thousands of web pages about and by the company.
Source
***
The inaccessible
nature of the Deep Web
may conjure images of hackers doing
nefarious things,
but that's not really accurate.
© aetb/iStock/Thinkstock
What a tangled web we weave, indeed.
About 40 percent of the world's population uses the Web for
news, entertainment, communication and myriad other purposes
[source:
Internet World Stats]. Yet even as more and more people log
on, they are actually finding less of the data that's stored
online.
That's because only a sliver of what we know as the
World Wide Web is easily accessible.
The so-called surface Web,
which all of us use routinely, consists of data that search
engines can find and then offer up in response to your queries.
But in the same way that only the tip of an iceberg is visible
to observers, a traditional
search engine sees only a small amount of the information
that's available - a measly 0.03 percent [source:
OEDB].
As for the rest of it...?
Well, a lot
of it's buried in what's called
the
Deep Web.
The Deep Web (also known as the undernet, invisible Web and
hidden Web, among other monikers) consists of data that you
won't locate with a simple Google search.
No one really knows how big the Deep
Web really is, but it's hundreds (or perhaps even thousands) of
times bigger that the surface Web. This data isn't necessarily
hidden on purpose. It's just hard for current search engine
technology to find and make sense of it.
There's a flip side of the Deep Web
that's a lot murkier - and, sometimes, darker - which is why
it's also known as the dark Web.
In the dark Web, users really
do intentionally bury data. Often, these parts of the Web are
accessible only if you use special browser software that helps
to peel away the onion-like layers of the dark Web.
This
software maintains the privacy of both the source and the
destination of data and the people who access it.
For political
dissidents and criminals alike, this kind of anonymity shows the
immense power of the dark Web, enabling transfers of
information, goods and services, legally or illegally, to the
chagrin of the powers-that-be all over the world.
Just as a search engine is simply
scratching the surface of the Web, we're only getting started.
Keep reading to find out how tangled our Web really becomes.
Hidden in Plain
Site
The Deep Web is enormous in
comparison to the surface Web.
Today's Web has more than 555
million registered domains. Each of those domains can have
dozens, hundreds or even thousands of sub-pages, many of
which aren't cataloged, and thus fall into the category of
Deep Web.
Although nobody really knows for
sure, the Deep Web may be 400 to 500 times bigger that the
surface Web [source:
BrightPlanet]. And both the surface and
Deep Web grow
bigger and bigger every day.
To understand why so much
information is out of sight of search engines, it helps to
have a bit of background on searching technologies. You can
read all about it with
How Internet Search Engines Work, but we'll give you a
quick rundown here.
Search engines generally create
an index of data by finding information
that's stored on Web sites and other online resources.
This
process means using automated spiders or
crawlers, which locate domains and then
follow hyperlinks to other domains, like an arachnid
following the silky tendrils of a web, in a sense creating a
sprawling map of the Web.
This index or map is your key to
finding specific data that's relevant to your needs. Each
time you enter a keyword search, results appear almost
instantly thanks to that index. Without it, the search
engine would literally have to start searching billions of
pages from scratch every time someone wanted information, a
process that would be both unwieldy and exasperating.
But search engines can't see
data stored to the Deep Web. There are data
incompatibilities and technical hurdles that complicate
indexing efforts. There are private Web sites that require
login
passwords before you can access the contents.
Crawlers
can't penetrate data that requires keyword searches on a
single, specific Web site. There are timed-access sites that
no longer allow public views once a certain time limit has
passed.
All of those challenges, and a
whole lot of others, make data much harder for search
engines to find and index.
Keep reading to see more about
what separates the surface and Deep Web.
Just Below the
Surface
If you think
of the Web like an iceberg,
the huge section below water is
the Deep Web,
and the
smaller section you can see above the water
is the surface
Web.
©Jan Will/iStock/Thinkstock
As we've already noted, there are
millions upon millions of sub-pages strewn throughout millions
of domains.
There are internal pages with no
external links, such as internal.howstuffworks.com, which are
used for site maintenance purposes. There are unpublished or
unlisted blog posts, picture galleries, file directories, and
untold amounts of content that search engines just can't see.
Here's just one example.
There are
many independent
newspaper Web sites online, and sometimes, search engines
index a few of the articles on those sites. That's particularly
true for major news stories that receive a lot of media
attention.
A quick Google search will undoubtedly unveil many
dozens of articles on, for example, World Cup
soccer teams.
But if you're looking for a more
obscure story, you may have to go directly to a specific
newspaper site and then browse or search content to find what
you're looking for. This is especially true as a news story
ages. The older the story, the more likely it's stored only on
the newspaper's archive, which isn't visible on the surface Web.
Subsequently, that story may not appear readily in search
engines - so it counts as part of the Deep Web.
Deep Potential
If we can
unlock the Deep Web to search professional databases
and
difficult-to-access deep information,
fields
such as medicine would immediately benefit.
©shironosov/iStock/Thinkstock
Data in the Deep Web is hard for
search engines to see, but unseen doesn't equal unimportant.
As you can see just from our newspaper example, there's
immense value in the information tucked away in the Deep
Web.
The Deep Web is an endless
repository for a mind-reeling amount of information. There
are engineering databases, financial information of all
kinds, medical papers, pictures, illustrations... the list
goes on, basically, forever.
And the Deep Web is only getting
deeper and more complicated. For search engines to increase
their usefulness, their
programmers must figure out how to dive into the
Deep Web and bring data to the surface.
Somehow they must not
only find valid information, but they must find a way to
present it without overwhelming the end users.
As with all things business, the
search engines are dealing with weightier concerns than
whether you and I are able to find the best apple crisp
recipe in the world. They want to help corporate powers find
and use the Deep Web in novel and valuable ways.
For example,
construction engineers could potentially search research
papers at multiple universities in order to find the latest
and greatest in bridge-building materials. Doctors could
swiftly locate the latest research on a specific disease.
The potential is unlimited. The
technical challenges are daunting. That's the draw of the
Deep Web.
Yet there's a murkier side to the
Deep Web, too
- one that's troubling to a lot of people for a lot reasons.
Darkness
Falls
The Deep Web may be a shadow
land of untapped potential, but with a bit of skill and
some luck, you can illuminate a lot of valuable
information that many people worked to archive.
On the
dark Web, where people purposely hide information,
they'd prefer it if you left the lights off.
The dark Web is a bit like
the Web's id. It's private. It's
anonymous. It's powerful. It unleashes human nature
in all its forms, both good and bad.
The bad stuff, as always,
gets most of the headlines. You can find illegal goods
and activities of all kinds through the dark Web. That
includes illicit drugs, child pornography, stolen credit
card numbers,
human trafficking, weapons, exotic animals,
copyrighted media and anything else you can think of.
Theoretically, you could even, say, hire a hit man to
kill someone you don't like.
But you won't find this
information with a Google search. These kinds of Web
sites require you to use special software, such as
The Onion Router, more commonly known
as Tor.
Tor is a software that
installs into your browser and sets up the specific
connections you need to access dark Web sites.
Critically, Tor is an encrypted technology that helps
people maintain anonymity online. It does this in part
by routing connections through servers around the world,
making them much harder to track.
Tor also lets people access
so-called hidden services - underground Web sites for
which the dark Web is notorious. Instead of seeing
domains that end in .com or .org, these hidden sites end
in .onion.
On the next below part we'll peel back the layers
of some of those onions.
Titillating Tor
In
October 2013, U.S. authorities shut down Silk
after the
alleged owner of the site Ross William Ulbricht was
arrested.
© David
Colbran/Demotix/Corbis
The most infamous of these onion
sites was the now-defunct Silk Road, an online marketplace
where users could buy drugs, guns and all sorts of other
illegal items.
The
FBI
eventually captured Ross Ulbricht, who operated Silk Road,
but copycat sites like Black Market Reloaded are still
readily available.
Oddly enough, Tor is the result
of research done by the U.S. Naval Research Laboratory,
which created Tor for political dissidents and
whistleblowers, allowing them to communicate without fear of
reprisal.
Tor was so effective in
providing anonymity for these groups that it didn't take
long for the criminally-minded to start using it as well.
That leaves U.S. law enforcement
in the ironic position of attempting to track criminals who
are using government-sponsored software to hide their
trails. Tor, it would seem, is a double-edged sword.
Anonymity is part and parcel on
the dark Web, but you may wonder how any money-related
transactions can happen when sellers and buyers can't
identify each other.
That's where
Bitcoin
comes in.
If you haven't heard of Bitcoin,
it's basically an encrypted digital currency. You can read
all about it on
How Bitcoin Works. Like regular cash, Bitcoin is good
for transactions of all kinds, and notably, it also allows
for anonymity; no one can trace a purchase, illegal or
otherwise.
Bitcoin may be the currency of
the future - a decentralized and unregulated type of money
free of the reins of any one government. But because Bitcoin
isn't backed by any government, its value fluctuates, often
wildly. It's anything but a safe place to store your life
savings.
But when paired properly with Tor, it's perhaps
the
closest thing to a foolproof way to buy and sell on the Web.
The Brighter
Side of Darkness
A
significant aspect of Bitcoin's appeal
is the
anonymity of transactions.
©
audioundwerbung/iStock/Thinkstock
The dark Web has its ominous
overtones.
But not everything on the
dark side is bad. There are all sorts of services that
don't necessarily run afoul of the law.
The dark Web is home to
alternate search engines,
e-mail services, file storage, file sharing, social
media, chat sites, news outlets and whistleblowing
sites, as well as sites that provide a safer meeting
ground for political dissidents and anyone else who may
find themselves on the fringes of society.
In an age where
NSA-type
surveillance is omnipresent and privacy seems like a
thing of the past, the dark Web offers some relief to
people who prize their
anonymity.
Dark Web search engines may not offer up
personalized search results, but they don't track your
online behavior or offer up an endless stream of
advertisements, either.
Bitcoin may not be entirely
stable, but it offers privacy, which is something your
credit card company most certainly does not.
For citizens living in
countries with violent or oppressive leaders, the dark
Web offers a more secure way to communicate with
like-minded individuals. Unlike Facebook or Twitter,
which are easy for determined authorities to monitor,
the dark Web provides deeper cover and a degree of
safety for those who would badmouth or plot to undermine
politicians or corporate overlords.
A paper written by
researchers at the University of Luxembourg (Content
and Popularity Analysis of Tor Hidden Services) attempted to
rank the most commonly accessed materials on the dark
Web.
What they found was that
although sites trading in illegal activities and adult
content are very popular, so too are those concerned
with human rights and freedom of information.
So although the dark Web
definitely has its ugly side, it has great potential,
too.
Even Deeper
The Deep Web is only getting
deeper.
Its store of human knowledge and trivialities
grows more massive every day, complicating our efforts
to make sense of it all. In the end, that's perhaps the
biggest challenge behind the Internet that we've
created.
Programmers will continue to
improve search engine
algorithms, making them better at delving into
deeper layers of the Web. In doing so, they'll help
researchers and businesses connect and cross-reference
information in ways that were never possible before.
At the same time, the
primary job of a smart
search engine is not to simply find information.
What you really want it to do is find the most relevant
information. Otherwise, you're left awash in a sea of
cluttered data that leaves you wishing you had never
clicked on that search button.
That's the problem of
so-called big data.
Big data is the
name for sets of data that are so large that they become
unmanageable and incoherent. Because the Internet is
growing so quickly, our whole world is overrun with
data, and it's hard for anyone to make sense of it all - even all of those powerful, all-knowing computers at
Bing and Google headquarters.
As the Internet grows, every
large company spends more and more money on data
management and analysis, both to keep their own
organizations functioning and also to obtain competitive
advantages over others. Mining and organizing the Deep
Web is a vital part of those strategies. Those companies
that learn to leverage this data for their own uses will
survive and perhaps change the world with new
technologies.
Those that rely only on the surface Web
won't be able to compete.
In the meantime, the Deep
Web will continue to perplex and fascinate everyone who
uses the Internet. It contains an enthralling amount of
knowledge that could help us evolve technologically and
as a species when connected to other bits of
information. And of course, its darker side will always
be lurking, too, just as it always does in human nature.
The Deep Web speaks to the
fathomless, scattered potential of not only the
Internet, but the human race, too.
Lots More
Information
Author's Note
The Deep Web is a vague,
ambiguous place.
But while researching this story, it
was easy to conclude at least one thing for sure - most
news headlines tend to sensationalize the dark Web and
its seedier side, and rarely mention the untapped
potential of the Deep Web.
Articles about illegal drugs
and weapons obviously draw more readers than those
detailing the technical challenges of harvesting data
from the Deep Web. Read the negative, breathless
articles with a grain of salt. It's worth remembering
that there's a whole lot more to the Deep Web than the
obvious criminal element.
As engineers find better,
faster ways to catalog the Web's stores of data, the
Internet as a whole could transform our society in
amazing ways.
Related Articles
Sources
-
Bagot, Martin. "Deep
Web Criminal Marketplace Completely Disappears
After '58 Million Heist'". The Mirror. Dec. 3,
2013. (Dec. 6, 2013) http://www.mirror.co.uk/news/technology-science/technology/sheep-marketplace-deep-web-criminal-2879995
-
Bergman, Michael K.
"White Paper: The Deep Web: Surfacing Hidden
Value." Journal of Electronic Publishing." Aug.
2001. (Dec. 6, 2013) http://quod.lib.umich.edu/cgi/t/text/text-idx?c=jep;view=text;rgn=main;idno=3336451.0007.104
-
Berman, Fran. "Deep
Web Film Will go Inside the World of Bitcoin,
Silk Road and Beyond." Mashable. Nov. 20, 2013.
(Dec. 6, 2013) http://mashable.com/2013/11/20/alex-winter-deep-web-documentary/
-
Biggs, John. "Deep
Web Users are Ready to Launch Silk Road 2.0."
Techcrunch. Oct. 4, 2013. (Dec. 6, 2013) http://techcrunch.com/2013/10/04/deep-web-users-are-ready-to-launch-silk-road-2-0/
-
Bingham, John and
Kirkup, James. "David Cameron Wins FBI Support
for 'Dark Web' War on Paedophiles." Telegraph.
Nov. 18, 2013. (Dec. 6, 2013) http://www.telegraph.co.uk/news/uknews/crime/10456108/David-Cameron-wins-FBI-support-for-dark-web-war-on-paedophiles.html
-
BrightPlanet.
"Understanding the Deep Web in 10 Minutes." Mar.
12, 2013. (Dec. 6, 2013) http://www.brightplanet.com/2013/03/whitepaper-understanding-the-deep-web-in-10-minutes/
-
BrightPlanet. "How
Does Data from the Deep Web go from Results to
Actionable Intelligence?" Jan. 31, 2013. (Dec.
6, 2013) http://www.brightplanet.com/2013/01/how-does-data-from-the-deep-web-go-from-results-to-actionable-intelligence/
-
BrightPlanet. "The
Future of Online Search." Aug. 27, 2013. (Dec.
6, 2013) http://www.brightplanet.com/2013/08/the-future-of-online-search/
-
Couts, Andrew. "TorSearch
Makes Finding the Next Silk Road a Lot Easier."
Digital Trends. Oct. 11, 2013. (Dec. 6, 2013)
http://www.digitaltrends.com/web/torsearch-tor-network-hidden-services/
-
Couts, Andrew. "That
Was Fast: 2 Sites Vie for Silk Road's Criminal
Customers." Digital Trends. Oct. 3, 2013. (Dec.
6, 2013) http://www.digitaltrends.com/web/silk-road-fbi-ulbricht-sheep-marketplace-bmr/
-
Gallagher, Diane.
"Peeling Back the Layers of the 'Dark Web' WCNC."
Nov. 19, 2013. (Dec. 6, 2013) http://www.wcnc.com/news/local/The-Dark-Web-232606871.html
-
Here and Now. "The
Deep Web: Where Google Won't Take You." WBUR.org.
Nov. 8, 2013. (Dec. 6, 2013) http://hereandnow.wbur.org/2013/11/08/the-deep-web
-
Hockerson, Lauren. "TorSearch
Offers Private Search of the Deep Web." Gigaom.
Oct. 11, 2013. (Dec. 6, 2013) http://gigaom.com/2013/10/11/torsearch-offers-private-search-of-the-deep-web/
-
Lederman, Abe and
Lederman, Sol. "Understanding Deep Web
Technologies." New Idea Engineering. Jun. 2004.
(Dec. 6, 2013) http://deepwebtech.com/PDFs/Understanding%20Deep%20Web%20Technologies.pdf
-
Love, Dylan.
"There's a Secret Internet for Drug Dealers,
Assassins and Pedophiles." Business Insider.
Mar. 6, 2013. (Dec. 6, 2013) http://www.businessinsider.com/tor-silk-road-deep-web-2013-3?op=1
-
Seymour, Andrew.
"Anonymous 'Deep Web' is New Frontier of Child
Exploitation, Conference Told." Ottawa Citizen.
Nov. 16, 2013. (Dec. 6, 2013) http://www.ottawacitizen.com/business/Anonymous+deep+frontier+child+exploitation+conference+told/9175718/story.html
-
Swift, Tim. "What is
the 'Deep Web'? And Other Questions About the
Shadowy Virtual World of Silk Road." Baltimore
Sun. Oct. 3, 2013. (Dec. 6, 2013) http://articles.baltimoresun.com/2013-10-03/business/bal-silk-road-deep-web-explainer-20131003_1_satoshi-nakamoto-bitcoin-silk-road
-
University of
California, Berkeley. "Invisible or Deep Web:
What It Is, How to Find It, and Its Inherent
Ambiguity." (Dec. 6, 2013) http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html
|
