Be a Search Bully!

First in a series...

Brief history of the Internet

What we know today as the Internet had is origins in the cold war. When the Soviet Union demonstrated it had the technological capability to launch an object into earth orbit by launching Sputnik in 1957 it was self-evident that the same technology could be utilized to launch an intercontinental ballistic missile. In response to this growing threat, the United States military began an ambitious project to maintain communication, command and control of its missiles and armed forces in the event of a devastating nuclear attack. It created what was known as the Advanced Research Projects Agency (ARPA), whose electronic communications and control plan became known as the ARPANET. The theory was that computers distributed over a large geographical area and networked together via redundant network connections would provide a safeguard against any one or more communications, command or control centers being “taken out” by a nuclear detonation. The first hard wired ARPANET connections were made in 1969. ARPANET was also referred to as the Internet.

Over the course of two decades, the Internet came to be used by both military and academic institutions alike. The World Wide Web was born upon the foundation of the Internet, which allowed for the transfer of public information on a global basis (as opposed to information exchange within the United States only). In this document, however, the word Internet will be given its colloquial meaning to include both the United States Internet and the World Wide Web.

By the 1990’s, growing public and private use of the Internet had spawned what is known today as the browser, a software application which allows for the navigation to, download and viewing of files, images and text via the Internet, whereas previously only text was viewable to the end user.

As use of the Internet grew, so did the amount of web pages and information contained therein. However, no one central authority maintained a “card catalog” of web sites. Navigation to useful information was most often done by word of mouth, via electronic mail, posts to virtual bulletin boards, and the like. Needless to say, finding what you were looking for was difficult at best. This gave rise to the search engine.

The first search engine was called “Archie”. Developed by a student at McGill University in Montreal in 1990, Archie worked by downloading the directory listings of files located on public servers. The database thus created by Archie was a database of file names, which were useful only insofar as the file name corresponded to the textual contents of the file. Archie did not, however, index the actual contents of the file. Therefore, if one was searching for information on the theory of relativity, Archie might not produce the relevant result of “Einstein.txt” unless the string “Einstein” was included in the search query.

An improvement on Archie was “Gopher”, developed by a University of Minnesota student in 1991 (and named after the school’s mascot). Gopher indexed the text contents of a file, providing much more information than the mere filename and expanding the usefulness of Internet searching. The text files indexed by Gopher were the precursor to modern web sites, albeit images were absent.

The Internet continued to grow and the need for automation became apparent. The first automated search bots appeared around 1993. These search bots would automatically spider and index web page contents into a searchable database. Eventually, these primitive search engines allowed for individuals or organizations to submit their own web sites to the index for crawling.

Notable additions to the search engine population were Excite (1993), WebCrawler and Lycos (1994), InfoSeek and Alta Vista (1995), Inktomi (1996), and Google (1997). Looking back at these search engines in their infancy, they bear little if any resemblance to their current form, both in appearance and functionality.

What is Search Engine Optimization?

With the advent of the modern search engines (as we now know them) in the 1990’s came problems surrounding the delivery of the organic search results – namely, how would it be determined which results would be delivered and in what order? As search engines are nothing more than software programs, they follow a set of specified rules to return the search results. This is known as the ranking algorithm.

As has been previously explained, search results were originally based upon file names (Archie) and then later file content (Gopher). Even with these primitive ranking algorithms, one can see how it is possible to manipulate the organic results served by these applications via manipulation the files – a primitive form of search engine optimization!

For example, if Archie was the only search engine around, a search engine optimization would consist of nothing more than naming your files properly. So, if I owned a search engine optimization firm (which I do), I would perform my service by first asking the following question: what keyword do you want to optimize your file for? (Note in this case we would be merely optimizing a file, more specifically a file name, as opposed to an entire web site, and as such would only have the opportunity to support one keyword).

Using the previous example of our file named Einstein.txt, let us say that we wish to actually optimize the file for the search query “relativity”. This would be because the file really has more to do with the theory of relativity than it does with the life and times of Albert Einstein. Because Archie’s ranking algorithm, which we are optimizing for, uses file names only, we are missing all kinds of traffic from people searching for “relativity”, and instead receiving traffic from people searching for biographical information on Albert Einstein. The tragedy is twofold: first, we are missing our targeted traffic; second, we are getting traffic from people who do not want what we have. It’s a double whammy! So, my advice would be to rename the text file relativity.txt – thereby getting rid of the elementary school children doing reports on Albert Einstein and attracting the graduate students we wish to target who are researching the theory of relativity.

This simple example demonstrates the basic concept behind search engine optimization – namely the manipulation of certain elements of a web site so as to “please” the search engine’s ranking algorithm. Unfortunately, these days it isn’t as simple as the example above.

Let us now advance to Gopher – which, instead of using file names actually does read the content of the file. Let us also, for the sake of this example, say that the Gopher ranking algorithm not only creates a raw database of search terms, but counts those terms and serves results based on the number of times that term appears in a document. Let’s call our modified Gopher program Gopher+. In Gopher+, therefore, a document containing the word “relativity” five times would rank higher than a document containing the same word only three times. As a search optimization engineer, I would advise the client to rewrite the content of the file to include the term “relativity” as many times as possible while still making grammatical sense. This would be the logical thing to do given the rather limited parameters of the ranking algorithm. Again, you can see how we are able to improve our placement in the organic search results via a manipulation of the file we want to get ranked. Especially discerning readers may already see the potential for abuse here.

If you are like me, you are not an “especially discerning reader”. So, what is the potential for abuse here?

In our example we are making the assumption that Gopher+ ranks pages according to the raw number of times a given search query appears within that document. Let us pretend that I own a web site about dogs. My web site contains the term “dogs” ten times. My competition, however, maintains a web site that contains the term “dogs” over 100 times, as they have written extensively on the subject. They have invested many hours and much money into developing their web site into a virtual fountain of information about dogs. I am too lazy to write enough text about dogs to incorporate that word over 100 times on my web site, or too greedy to pay somebody else to do it, yet I desire to get all the traffic that is going to my competition’s web site.

What I decide to do is this: at the bottom of my web site, I type the following text, after the main text (or body) of the web site:

dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs dogs

By my count, that is 224 uses of the word “dogs” on my web page, plus the original 10 times I used it in the main body of the text, for a grand total of 234 occurrences of the word “dogs”. I have just accomplished the following:

• I have made little to no additional investment in my web site
• I have added no significant or useful content or information to my web site
• Yet I have surpassed my competition’s use of the word “dogs” despite the fact that they have made a substantial investment in their web site
• And I have defrauded Gopher+ into ranking my site as more relevant in its organic results

This is a known spamming technique, and will result in your site being severely penalized or banned from any respectable search engine.

As we progress in our study we will learn about additional ways to manipulate the ranking algorithms of search engines. It is important to remember that the manipulation of the ranking algorithms is not a bad thing in and of itself – in fact it is quite desirable, and is known as making a site search engine friendly. It is only when such manipulation involves spam – dishonesty, deceptiveness, or fraud – that we wish to avoid it like the plague.

Making a site “search engine friendly”

You may have heard of the term “search engine friendly”. Making a web site search engine friendly is an integral part of any quality search engine optimization project. While the difference between the terms “search engine friendly” and “search engine optimization” may be esoteric (and to some seem like splitting hairs), there is a difference. A website may be optimized but not be search engine friendly (for example, keyword and content rich, but impossible to navigate), likewise a web site may be search engine friendly but not be optimized (for example, by having a dearth of relevant keywords).

Search engine friendliness encompasses primarily ease of navigation and readability of the code and site content. Keep in mind that I am using these terms from the perspective of a search bot – not from that of a human. A web site that appears to be easily navigable to the human eye may be difficult or impossible for a search engine to navigate, for example. We will delve further into this later on.

The second primary aspect of search engine friendliness is compliance with a search engine’s Terms of Inclusion which, succinctly put, means the avoidance of spam.

It is not so important to be able to academically describe the difference between the terms search engine friendliness and search engine optimization – it is only important to know that any thorough search engine optimization project will contemplate search engine friendliness. So as not to get bogged down in academics, however, for our purposes we will use the terms search engine friendliness and search engine optimization interchangeably. Do not be alarmed, however, as you may rest assured all bases will be covered.

What Search Engine Optimization is not

Many web site owners these days are getting caught up in the web (no pun intended) of pay per click advertising. We have already discussed the fact that search engine optimization is concerned with increasing a web site’s ranking in the organic search results. As such, search engine optimization does not encompass PPC schemes. In fact, in my humble opinion, the entire point of search engine optimization is to avoid the recurring and potentially infinite charges involved with paid inclusion.

I have had client after client complain about recurring PPC charges in the thousands of dollars per year or even per month! I explain to them that all this could be avoided with a one time search engine optimization that will place their site in the organic search results without the need for recurring expenses.

Paid inclusion can be an important part of a marketing and advertising strategy for either organizations with large budgets or for a company making its Internet debut. Oftentimes I will recommend a limited paid inclusion program to my clients for a period of 60 to 90 days after a search engine optimization, after which the paid inclusion program is stopped. The reason for this is that there can be a bit of delay between the completion of a search engine optimization and the realization of the results.

By far the most successful paid inclusion programs are Google’s AdWords, adwords.google.com and Yahoo!’s Search Marketing, www.overture.com (formerly known as Overture). Generally these models work in a manner by which advertisers bid for placement on certain keyphrases, with the higher bids being placed above the lower bids. The problem with this is that there is no ceiling, and the competition for first position drives the bid price skyward over time, meaning you will either have to increase your budget or decrease the amount of click-throughs you receive, neither of which is an attractive option. Also, conversion rates for these models hover under 5% on average, which means at very best you are likely to receive one purchase for every twenty click-throughs. This may be fine if you are selling a large ticket item and the bid price is relatively cheap, however for small ticket items it can quickly become a money sink.

Search engine optimization is also not an overnight process in many cases. As explained above, results can take as long as 90 days, and sometimes longer, to materialize. Oftentimes, however, some improvement in organic search results is noticed almost immediately. It is always fun to track the progress of your site. I like to record a web site’s position for each of its keyphrases prior to the commencement of a search engine optimization project, then track its improvement over time. For more information on how to do this, see the section entitled Step 8: Maintenance and Tracking.

Nor is search engine optimization a substitute for quality content, as we briefly demonstrated with the dog example earlier. In fact, quality content is integral to achieving the high rankings you desire. We will study exactly what makes content desirable to a search engine at a later date, but as an example, a web site that says nothing more than “Welcome to my home page. My name is John Doe.” really has nothing to offer the human visitor or the search engine. Your web site should say something – about you, your company, your organization, your mission, or your product. It really must have some sort of useful, quality information (content) from which the user will benefit.

Finally, not to beat a dead horse, but search engine optimization is not spam. It is, in fact, the avoidance of spamming techniques. Pay attention to the spam warnings located in these lessons, as there are many disreputable companies out there who will gladly take your money, purporting to sell you a “search engine optimization”. All they will really do is engage in some sort of spamming technique (we will talk about all the major ones) which may achieve a temporary spike in organic rankings but will likely within a few weeks result in severe penalties or permanent banishment. If all the ethical arguments fail, it is simply not financially worth it.

Why Search Engine Optimization?

Prior to the dawn of the Internet, when a person was in search of something they usually picked up the Yellow Pages. The Yellow Pages was a convenient way to search listings of “brick and mortar” business by business category. Of course, the Yellow Pages did not rank businesses in any way and the Internet analogy would be a paid inclusion model. In other words, the Yellow Pages would list anyone who had the money to be listed, regardless of the relevancy of the listing to any particular category.

Building a web site without making an investment in search engine optimization is analogous to building a brick and mortar business with no windows, no signage, no advertising, and an unlisted telephone number. Your business is there, but there is no way for anyone to discover it. Simply put, the establishment of a web presence is simply not enough.

Search engines allow your customers to find you. The wonderful thing about search engines is that they drive targeted traffic to your site – these are customers that have already told the search engine that they are looking for the products or services that you offer! Contrast this with television, radio or print advertising – which do not effectively target your particular audience, but rather utilize a “shotgun” approach.

You may be asking whether or not anyone actually uses search engines to find what they are looking for. After all, don’t most people still rely on the Yellow Pages, television, radio, or other print media? The answer to that is not only no, but “Heck, no!”

As can be easily seen from the above graphic, neglecting the power of search engine marketing can be a very costly mistake, costing you almost half of your customers! In fact, it is generally accepted among experts in the field that it costs you far more not to invest in search engine optimization. Search engine optimization is the single most important investment you can make in your web site, aside from the web site itself. In fact, I would venture that it is more important than the web site itself. An inexpensive web site with limited graphical content can be easily developed, and with proper search engine optimization can generate sales far beyond an expensive, multimedia web site which is not properly optimized. If you are on a limited budget and must cut corners, cut corners on the design aspect of the web site. It is simply too costly to cut corners on search engine optimization.

As the Internet grows, so do the databases of the search engines. In 2003, several search engines announced that their databases had exceeded the 3 billion mark in terms of items indexed. This is an awesome number. The only way to stand out in such a large crowd is to be at the top. You may have the best product in the world, but if nobody can find it, it is virtually without value. Perhaps you are getting the idea - I simply cannot overemphasize the need for quality search engine optimization!

We have already learned of two historical ranking algorithms, based upon either the name of a file (Archie) or the content of a file (Gopher). Other ranking algorithms utilize different methods, or a combination of methods.

The META Tag Myth

So many times I speak with clients who say something to the effect of, “I don’t see what the big deal is. All you have to do is go in and fix the META tags and the site is optimized.” That couldn’t be further from the truth!

A brief description of the structure of a basic web page is in order at this time. I understand that to the untrained reader this section may seem a bit technical. Please don’t worry – and please keep reading - you are not expected to learn or know HTML, that is not within the scope of this study. Simply try to grasp the concepts being explained, without worrying about the minutia of the code snippets below.

An HTML file consists of two basic sections, the HEAD and the BODY. The HEAD section contains snippets of code that are for the most part invisible to the user, but that do contain information about the web page, such as the author, date of creation, and relevant to our discussion the KEYWORD and DESCRIPTION META tags, and the TITLE tag. The BODY section of an HTML file consists of the actual code which governs the display and functionality of the web page – in other words, the part that you see.

Early on META tags were a useful tool for search engines, as the search engine could simply read the META contents of a file and then tell what the file contained. For example, a web page for a small business that rents inflatable bouncers for children’s parties and is located in La Jolla, California would want to include the following information in its HEAD tags:

title>Inflatable bouncer rentals La Jolla/title>
meta name=”keywords” content=”inflatable bouncer bouncers moonwalk moonwalks rent rental rentals La Jolla”>
meta name=”description” content=”La Jolla, California area inflatable bouncer rentals for children’s parties or corporate events.”>

A search engine need look no further than the HEAD tags of the web page to index the page. It is quick, easy and efficient, and does not require sorting through the entire contents of the web page to determine what the web page might be about. The inherent problem in this is, of course, human nature. This method relies on a human being, specifically a human being with a vested interest in driving traffic to his or her web site, to tell you what the web site is about. The problem of keyword stuffing again arose. For example, the above business might use the following KEYWORDS tag instead of the proper one given above:

meta name=”keywords” content=”inflatable bouncers inflatable bouncers inflatable bouncers inflatable bouncers”>

This being done, of course, to make the search engine think the web site was more about inflatable bouncers than another site of equal importance. So we have the same problem as we did before, with the dog example! But it gets worse – what if the web site owner were to include keywords that had nothing to do with his or her web site but were extremely popular keywords? This would be done in an attempt to drive raw traffic to the web site, without regard for the fact that the people making the searches have no interest in what is being sold on the site. If the same inflatable bouncer business did this, it might take this form:

meta name=”keywords” content=”sex mp3 sex mp3 sex mp3 sex mp3 inflatable bouncers”>

A person doing this is fooling both the search engine and the end user into thinking the web site concerns a topic entirely unrelated to the contents of the web site. Another deceptive practice is as follows:

meta name=”keywords” content=”Apple Computer Corporation MacIntosh”>

There is no problem with the above until I tell you this: suppose this tag appeared in the header of the web site at www.ibm.com? The idea would be to lure people who were searching for Apple products onto a competitor’s web site in an effort to steal the business. Same could be done for Coke vs. Pepsi, Chevy vs. Ford, or Joe’s General Store vs. Sam’s General Store.

Because of the spamming of the KEYWORD META tag, today all major search engines completely ignore this tag. It might as well be left out of the HTML file. However, because clients always seem to get antsy if this tag is left out, I do include it as a “pacification device”, so to speak. It achieves, however, nothing (but it does not hurt the client, either. It is just, literally, nothing.)

This still leaves the TITLE and DESCRIPTION tag, though. These tags, in and of themselves, accomplish nothing for a web site. Due to the same concerns as are present with the KEYWORD tag, these tags are disregarded in their own right. However, and this is a big however, these tags are used by search engines to ensure that a web site is not engaging in spamming activity. This is done via a process known as META-Content Matching.

META-Content Matching is exactly what it seems. A search engine takes the content in the HEAD tags, for instance “Apple Computer Corporation MacIntosh”, and checks the actual content of the web page to make sure those terms are utilized in the body of the web page. If the HEAD content does not match the actual content, a spam flag is raised and the web site can be penalized or banned. Thus, it is quite important to make sure that the HEAD content accurately reflect the actual content of a web site. It is only in this regard that the TITLE and DESCRIPTION tags gain importance today. If the content matches up, then these two tags are given weight by a search engine. We will discuss HEAD tags more in depth in Step 4: Building Relevancy: Optimization of HTML Head Tags.

Relevancy Algorithms Based on Site Content

Another way search engines rank sites is by the content of the site, as we discussed in simple form using the Gopher and Gopher+ examples. In a pinch, the one thing that is most important to obtaining and maintaining high search rankings is the content of the website. But what constitutes content?

We already know, thanks to that student in Minnesota, that the actual text written on a web page constitutes site content. What we might not have considered is other forms of content – such as images, multimedia and the like. Basically, the content of a web site can be considered to be anything that is presented to the end user in any form – be it visible on the screen, audible to the ears, or downloadable to the desktop. All types of content are relevant to the organic search results, the challenge is getting the search engines to read and understand what the content is. Exactly how to go about achieving this is the primary focus of our study, and will be discussed throughout.

Relevancy Algorithms Based on Site Backlinkage

Links to or from a site form another basis for ranking a web site. This is especially true with Google, in fact in its original form Google ranked web sites almost solely based upon the number of sites linking to that site. It can be visualized as sort of a “virtual referendum”, with each link to a web page counting as one “vote” in favor of that web page. Site backlinks continue to play an important role in search engine optimization, but only if done correctly, and we will discuss the issue fully in Step 6: Maximizing Relevancy: Link Popularity.

Relevancy Algorithms Based on Site Freshness

The most recent addition to the search engine ranking family are algorithms based upon fresh, or new, content. Dynamic web sites, sites that are frequently updated, and sites that provide new and exciting content rank higher than static sites. Step 7: Over the Top: Adding Fresh and Interactive Content will take us through all the latest and greatest tools and technologies including weblogs, RSS and XML newsfeeds, and content syndication.

Modern Hybrid Relevancy Algorithms

Modern search engines do not serve their organic search results based upon any one of the algorithms elucidated above. Today’s search results are delivered via vastly complex formulae which take into account many or all of the factors we have mentioned so far, assigning various weights to each of them to determine a result. The best analogy I can think of is the ranking of college football teams by the Bowl Championship Series (BCS). The final rankings are based not on any one factor (such as win/loss record, total points scored, total offense, total defense, coaches poll results, Sagarin rankings), but rather is a hybrid combination of it all, each being given its own level of importance. Today’s search engine optimization, therefore, is an all-encompassing, far-reaching, “cover-all-your-bases” type of science. Don’t be sold by anyone trying to sell you only one portion of the deal, such as backlinks, or META tags, or the like. Done properly, search engine optimization is an exhaustive yet extremely effective endeavor.

But Which Search Engine Do I Optimize For?

The quick answer? ALL OF THEM. How do we do that? By doing what we just said above – optimizing everything, every single possible thing that could be used by a search engine algorithm to establish a site’s ranking.

Interestingly enough, most search engines today derive their results based upon three major players: Google, Yahoo!, and the Open Directory Project. Therefore, if we pay special attention to obtaining quality results on these three engines, we can virtually be assured of obtaining similar results on the rest of them!

Should I Hire an SEO Firm?

Unless you have at least some basic technical knowledge, i.e. the ability to publish your own web site using Dreamweaver or similar software, probably so. This study will allow you to optimize your web site yourself if you have some technical knowledge (or have published a web site before) or if you have an in-house IT guy (or gal) that just needs to know what to do. If you can’t do it yourself, what this study will do is arm you with the powerful knowledge of what effective search engine optimization means, so that you will be able to select a search engine optimization firm from a position of knowledge, rather than ignorance. Oftentimes in this field, disreputable firms will take advantage of people’s ignorance and sell them snake oil, which can end up actually harming the client’s business. Use the knowledge you gain from this study to empower yourself – and if after reading this you know more than the firm you are considering hiring, it is probably time to find another firm.

Factors to consider when hiring an SEO firm include:

• Are they knowledgeable about what is needed to effect a total solution, or is only a partial solution or “band-aid” being offered?
• Do they engage in known spamming techniques (everyone will tell you no, but your knowledge of them through this study will help you determine that for yourself)
• Price should not be the deciding factor when it comes to SEO – quality should
• Be very wary of “guaranteed top rankings” – nobody can guarantee a first place ranking
• Are recurring fees a part of the payment structure? If so, what are they for? Shy away from paid “monitoring” or “reporting” services – you can do that yourself by simply performing your own searches for yourself. Only agree to recurring fees if actual, ongoing work is being done, outside of monitoring or reporting.