In what year did search engines appear? Search engines. Search Engine Market

In the early days of the Internet, users were a privileged minority and the amount of information available was relatively small. At that time, access to it was mainly available to employees of various large educational institutions and laboratories, and the data obtained were used for scientific purposes. At that time, the use of the Internet was not as relevant as it is now.

In 1990 British scientist Tim Berners-Lee (who is also the inventor of URI, URL, HTTP, World Wide Web) created the site info.cern.ch, which is the world's first accessible directory of Internet sites. From that moment on, the Internet began to gain popularity not only among scientific circles, but also among ordinary owners of personal computers.

Thus, the first way to facilitate access to information resources The formation of website directories has begun on the Internet. Links to resources in them were grouped by topic.

The first project of this kind is considered to be Yahoo, open in April 1994. Due to the rapid growth in the number of sites in it, it soon became possible to search for the necessary information upon request. Of course, it was not yet a full-fledged search engine. The search was limited to only the data that was in the catalog.

In the early stages of the development of the Internet, link directories were used very actively, but gradually lost their popularity. The reason is simple: even if there are many resources in modern catalogs, they still show only a small part of the information available on the Internet. For example, the largest directory on the network is - DMOZ(Open Directory Project). It contains information on just over five million resources, which is dwarfed by Google's search database of over eight billion documents.

The largest Russian-language catalog is the Yandex catalogue. It contains information about just over one hundred and four thousand resources.

Timeline of search engine development

1945- American engineer Vannevar Bush published a record of the idea that later led to the invention of hypertext, and a discussion of the need to develop a system for quickly retrieving data from information stored in this way (the equivalent of today's search engines). The concept of a memory expander device he introduced contained original ideas that eventually came to fruition on the Internet.

1960s— Gerard Salton and his team at Cornell University developed the SMART information retrieval system. SMART is an abbreviation for Salton’s Magic Automatic Retriever of Text, that is, “Salton’s Magic Automatic Text Retriever.” Gerard Salton is considered the father of modern search technology.

1987-1989 – developed Archie— a search engine for indexing FTP archives. Archie was a script that automated the implementation of listings on ftp servers, which were then transferred to local files, and only then carried out in local files quick search necessary information. The search was based on a standard Unix grep command, and user access to the data was based on telnet.

In the next version, the data was divided into separate databases, one of which contained only text file names; and the other - records with links to hierarchical directories of thousands of hosts; and another one connecting the first two. This version of Archie was more efficient than the previous one, since the search was carried out only by file names, eliminating many pre-existing duplicates.

The search engine became more and more popular, and the developers began to think about how to speed up its work. The database mentioned above has been replaced by another one based on compressed tree theory. A new version, essentially created a full-text database instead of a list of file names, and was significantly faster than before. In addition, minor changes allowed the Archie system to index web pages. Unfortunately, for various reasons, work on Archie soon ceased.

In 1993 The world's first search engine was created World Wide Web Wandex. It was based on the World Wide Web Wanderer bot developed by Matthew Gray from the Massachusetts Institute of Technology.

1993– Martin Koster creates Aliweb- one of the first search engines on the World Wide Web. Site owners had to add them to the Aliweb index themselves in order for them to appear in searches. Because too few webmasters did it, Aliweb did not become popular

April 20, 1994– Brian Pinkerton from the University of Washington released WebCrawler— the first bot that indexed pages completely. The main difference between the search engine and its predecessors is that it allows users to search for any keyword on any web page. Today this technology is the search standard of any search engine. The WebCrawler search engine became the first system that was known to a wide range of users. Alas throughput was low and the system was often unavailable during the daytime.

July 20, 1994– opened Lycos is a major development in search technology created at Carnegie Melon University. Michael Maldin was responsible for this search engine and still remains a leading specialist at Lycos Inc. Lycos opened with a catalog of 54,000 documents. And in addition to this, the results it provided were ranked, and it also took prefixes and approximate matches into account. But the main difference between Lycos was its constantly expanding catalog: by November 1996, 60 million documents were indexed - more than any other search engine at the time.

January 1994- was founded Infoseek. It wasn't truly innovative, but it had a number of useful additions. One of these popular additions was the ability to add your page in real time.

1995- started AltaVista. Having appeared, the AltaVista search system quickly gained recognition from users and became a leader among its peers. The system had virtually unlimited bandwidth at that time, it was the first search engine in which it was possible to formulate queries in natural language, as well as formulate complex queries. Users were allowed to add or remove their own URLs within 24 hours. AltaVista also offered many search tips and tricks. The main merit of the AltaVista system is considered to be support for many languages, including Chinese, Japanese and Korean. Indeed, in 1997, not a single search engine on the Internet worked with several languages, especially rare ones.

1996— the AltaVista search engine has launched a morphological extension for the Russian language. In the same year, the first domestic search engines were launched - Rambler.ru and Aport.ru. The emergence of the first domestic search engines marked a new stage in the development of the Runet, allowing Russian-speaking users to make queries in their native language, as well as quickly respond to changes occurring within the Network.

May 20, 1996— the Inktomi corporation appeared along with its search engine Hotbot. Its creators were two teams from the University of California. When the site appeared, it quickly became popular. In October 2001, Danny Sullivan wrote an article entitled "Inktomi's Spam Sites Database Is Open to the Public," which described how Inktomi accidentally made its database of spam sites, which by then contained about 1 million URLs, available to the public. general use.

1997– in Western countries there comes a turning point in the development of search engines, when S. Brin and L. Page from Stanford University founded Google(the original name of the BackRub project). They developed their own search engine, which gave users the opportunity to carry out high-quality searches, taking into account morphology, errors in spelling words, and also increase relevance in query results.

September 23, 1997– announced Yandex, which quickly became the most popular search engine among Russian-speaking Internet users. With the launch of the Yandex search engine, domestic search engines began to compete with each other, improving the system of searching and indexing sites, delivering results, as well as offering new services.

Thus, the development of search engines and their formation can be characterized by the stages listed above.

Today, three leaders have settled in the global market - Google, Yahoo and Bing. They have their own databases and their own search algorithms. Many other search engines use the results of these three major search engines. For example, AOL uses a database Google data while AltaVista, Lycos and AllTheWeb use the Yahoo database. All other search engines in various combinations use the results (issues) of the listed systems.

If we conduct a similar analysis of search engines popular in the CIS countries, we will see that mail.ru broadcasts Google search, while applying its new developments, Rambler, in turn, broadcasts Yandex. Therefore, the entire RuNet market can be divided between these two giants.

That is why, in the CIS countries, website promotion, as a rule, is carried out only in these two PS.

1. Introduction

Search engine optimization of websites usually includes transforming the structure of the website, filling it with text content depending on the queries being promoted, as well as various ways increasing the citation of the resource. In addition to standard optimization processes, there are numerous methods that are already applied to a specific search engine. Of course, search engines open source is still a rarity, and optimizers have a lot to puzzle over, but some basic patterns of behavior can be traced in already established systems.

In this master class we are with a touch comparative analysis Let's look at the most popular foreign search engines and Russian search engines.

2. Russian search engines

Google, Yandex, and Rambler are currently considered the most popular search engines in Russia. The Yandex system is more visited and has relevant results to the required extent, therefore, most often, optimizers undertake to promote the resource in this system.

Let's take a closer look at each of these search engines.

Google

Google has its own Russian analogue, less popular than the original American one, but in no way inferior to it in terms of search quality. Google has numerous fans in Russia who believe that this search engine produces the best results.

PageRank used by Google is primarily based on link popularity. Those. When calculating page relevance, the biggest contribution comes from the number and quality of links to pages from other pages.

On this moment Google's database contains more than a billion indexed pages.

Google is one of the few search engines that deeply indexes your site. Google uses link popularity as the most significant factor in determining a page's relevance. Therefore, it is easier for large and popular sites to rank high in search results. This also protects Google from spam.

Google believes that the following factors will be most influential in ranking pages:

• Link popularity (citation index);
• Keyword proximity and density (density and frequency of keywords);
• Keywords in the link text (keywords in links);
• Emphasized text (emphasized text).

Distinctive feature Yandex – intuitive search in all word forms. A unique development for the peculiarities of the Russian language.

The Yandex catalog uses such a concept as thematic citation index (TCI) . It is determined by the number and quality of external links to your site. The quality of the link is affected by the TCI of the resource linking to you. The thematic proximity of your site to the resources linking to you plays an important role.

When calculating the citation index, links from forums, web boards, conferences, and sites located on free hosting(if they are not described in the Yandex catalog). Naturally, links from sites that Yandex does not index (for example, foreign sites) are not taken into account.

The number of hosts depends on visitors (the more there are, the more hosts), and the Yandex citation index depends on the site authors (the more authors link to your resource, the higher the CY value).

The value of the citation index determines the relevance of the resource in the Yandex catalog and, accordingly, the position of your site in the selected section.

Document re-indexing occurs approximately once every two weeks, but the robot is adjusted to each site separately. It all depends on the update frequency. According to Yandex, the tag< meta name="Revizit-after" content="n-days">does not play any role in the operation of the robot.

Yandex indexes the Russian network, so servers in the domains su, ru, am, az, by, ge, kg, kz, md, ua, uz are entered into the search engine. The remaining servers are included only if text in Russian is found on them.

Header information (tag< title >) Yandex displays in search results. Words in the tag< title >, have more weight than everyone else. Keywords in the tag< meta>also increase the weight of a word in a document, but only if the word itself is on the page.

In addition to the above methods, the relevance of a word is influenced by the frequency of its use in headlines (< h1>, < h2>...), in the alt attribute, in tooltips (tag< acronym>) and the percentage of occurrence of this word in the document, i.e. how often do you use it. But at the same time, it is necessary to preserve the meaning of the document, otherwise Yandex may consider this word spam.

Rambler

It is the best rubricator today. It is convenient because all resources are divided into sections, subsections, etc. Many of them participate in the Top100 rating, which allows us to determine the popularity of a particular resource.

It is believed that the fundamental factor for website promotion in Rambler is good content and traffic, measured using a counter installed on the site. Thus, high positions in the ranking can be occupied by a well-promoted, high-quality and long-established site on the Internet, with high traffic.

Aport

The least popular of the listed search engines. The main criteria that this system takes into account when sorting sites:

• the number of search words in the document text (in percentage), the distance between search words in the document text;
• the place in the text where the search words occur (title, description, meta tag, etc.);
• appearance font in which the search words are typed in the text (size, boldness, color);
• number of links from the Internet to this document;
• use of the search word in the text of links from the Internet to this document.

3. Foreign search engines

The foreign search engines presented below are widely known and used all over the world. For webmasters, these search engines are the most important places to register, as they have the potential to drive a large number of visitors to the site. Some of the listed systems are not full-fledged search engines, but only use the resources of other well-known search engines.

It is considered one of the largest search engines based on the number of indexed pages. The system gained great popularity due to the ability to search using complex selection criteria. AltaVista offers additional services in the form of searching directories from the Open Directory, LookSmart, Ask Jeeves.

Google is a search engine that uses the number of links to a website as its main measure of a site's popularity. This is especially useful when searching for good sites using simple search queries. Google is famous for highly relevant links. Google has a very large database of indexed sites and provides some of its results to Yahoo and Netscape Search.

Yahoo is the most popular and oldest search engine. Yahoo employs about 150 editors who compose and edit the content of its directories. Yahoo has over 1 million indexed sites in its database. Also, in case of lack of its own database, Yahoo uses the Google database, and previously, it must be said, it used the Inktomi database.

Direct Hit measures the number of clicks. Sites that get more clicks get best rating. Known as the "popularity search engine".

Direct Hit results also appear in HotBot and can also be optionally shown in MSN Search.

System results are taken from LookSmart, Inktomi, RealNames and Direct Hit. Search users have a unique ability to save previous search results.

A striking difference of the system is that positions in the search results are sold. GoTo considers this to be more relevant. It is also possible to buy higher seats in various ratings. Unpaid results are provided by Inktomi.

This system does not implement the ability to enter a search word directly, only through the interface of other search engines. And there are a great many of them.

A characteristic feature can be called excellent results for each of their partners in this base. Varying the results allows the results of a particular search engine to differ from others.

This system was filled with the help of volunteers. After sufficient formation, it became possible to freely use its results. Currently, Open Directory is used by Lycos, AOL Search, AltaVista and HotBot.

A distinctive feature of this system is that when you type a keyword (for example, a company brand), you can get to the official website of that company.

4. Features of promotion

Whatever the search system, Russian or foreign, whatever filtering algorithms it has, the main criterion for assessing the relevance of a resource for a specific request is the presence of high-quality text content, which is good news. For conscientious optimizers, the primary task when promoting a resource is, first of all, optimizing the site’s content, its internal link navigation and convenience for the direct user, and not direct optimization for a specific search engine.

The second most important step when promoting in many search engines is having the maximum number of links to a resource. Although there is an opinion among many direct users that this criterion cannot fully provide the result, today it is necessary.

We can say that other features of promotion for different search engines, which, frankly speaking, are not entirely “clean”, are formed directly by the costs of the filtering algorithms of the search engines themselves. Thus, the search engines themselves provoke optimizers to use tricks. This is especially true for the main Russian search engines, which are characterized by a tendency towards monopolism.

5. Conclusion

The master class allows you to evaluate the fundamental differences between search giants and get some idea of ​​how you can get more or less relevant information on the Internet. The presence of numerous search engines creates significant competition, and, as we know, competition entails processes aimed at progress and development in the right direction. In this case, the main priority is still a quality result. Hooray!

In the early years of the development of the Internet community, active Internet users were a minority, and the amount of information on Internet resources was relatively small. Access to the world information network For the most part, only employees of scientific laboratories and large educational institutions had. In general, the use of a network resource was not as relevant as it is today.

History of the development of search engines

A big step towards spreading the Internet to the masses was the appearance in 1990 of the website info.centr.ch. This site was the first publicly accessible directory of Internet sites. The creator is British scientist Tim Berners-Lee, who is also considered the creator of URI, HTTP, World Wide Web and URL. Since that moment, Internet sites have become relevant not only in specialized circles of users, but also among ordinary owners of home computers. In this catalogue, for convenience, information resources were arranged based on groups on similar topics, which greatly facilitated the search for information.

But progress did not stop there, and in 1994, a search technology developed by Carnegie University, known as Lucos, was born. This directory, which was created by Michael Maldin, started with a resource of more than 50,000 documents. In Lucos, queries took into account approximate matches of the query, and the search result was ranked depending on the match between the input and output information. And also, the resource was constantly updated with new Internet pages. In November, Lucos already had more than 55 million pages and documents, far more than any document catalog at that time.

At the end of 1994, the Infosek resource appeared. It had a number of advantages relative to other resources. For example, adding sites by a user to a directory database in real time.

The new monster in the search engine industry in 1995 was AltaVista. She quickly gained popularity among Internet users and took a leading position in her field. Its main feature was the ability to formulate queries in natural, colloquial language, and users were also allowed to add their own URL addresses. But still, the main merit of AltaVista was its support for multiple language packs, such as Korean, Japanese, and Chinese, as well as Russian.

A huge step in search technology was the appearance on the Internet of a new search engine, the name of which is now well known to every user, namely Google. In 1997, L. Page and S. Brin from Stanford University introduced new features into the search algorithms of their brainchild. The search used systems of relevance of the search results produced by the system, and when making a request, morphology and possible spelling errors were taken into account.

Nowadays, there are three main leaders in the search engine market - Bing, Google and Yahoo. They have at their disposal search algorithms and databases of their own production. Many search engines, of which there are many, use the developments of these three titans among search engines.

Thanks to search engines, it has become easier for the common person to discover the vast expanses of the information field. Without their development, it is impossible to improve the ways of exchanging information between people.

At the time when the development of the Internet was just beginning, the amount of available information was relatively small, and there were few Internet users. In the initial stages of the network's development, it was used by employees of universities and research laboratories to exchange information between institutions. At that time, searching for information on the Internet was not relevant, unlike today.

The first way to organize and systematize access to information resources was the creation of website directories. They began to group links according to specific topics.

The pioneer in the area was Yahoo, which appeared in April 1994. Over time, the number of sites increased and Yahoo added a directory search option. It was not a search engine in the truest sense, because the search scope was limited to directory resources.

Subsequently, catalogs spread and began to be used everywhere, but the Internet did not stand still, but continued to develop. Search methods also developed along with it. At the moment, directories have almost lost popularity, this is explained by the fact that a modern directory, even if it contains a huge amount of resources, can only provide access to a small part of the information contained on the network.

These days, the largest directory on the web is the Open Directory Project, or DMOZ, which contains information on 5 million resources, but this is relatively small when compared to, for example, Google's search engine, which contains about 8 billion documents.

A full-fledged search system was released only in 1994, it became the WebCrawler search system.

A year later, in 1995, the AltaVista and Lycos search engine projects appeared. One of them, AltaVista, in particular, held a leading position in the search field for many years.

Two years later, in 1997, Stanford University students Sergey Brin and Larry Page developed the Google search engine, which is the leader in search today.

This year was also the year when the creation of the Russian search engine Yandex was officially announced, which is still the leader in the Russian-language segment of the network.

At the moment, there are only 3 search engines that have reached the international level: MSN Search, Yahoo and Google. These systems have their own databases and search algorithms. Most other search engines use their results. So Mail.ru uses the Yandex database, search.aol.com uses Google, and Lycos, AltaVista and AllTheWeb use Yahoo.

The search leader on the Russian Internet at the moment is Yandex, followed by Rambler, followed by Google, Mail.ru, A port and KM.ru.

Search engines have different operating algorithms, and in order to take a good position in search results and attract targeted visitors, you need to know the features of SEO optimization for different search engines. For example

It is generally accepted that the history of the first search engines in the Russian segment of the Internet begins in 1995. It was this year that a morphological extension to the Altavista search engine became available to Runet users. Almost immediately after the expansion, the original search engines Aport and Rambler appeared, which are considered the first Russian search engines.

AltaVista was released in December 1995 and was supported by the most powerful computing server available at the time, DEC Alpha. It was the fastest search engine that could handle millions of search queries per day.

Aport

The Aport search system was demonstrated to the general public several months before Rambler in February 1996. At the time of its launch, the machine searched only on the site russia.agama.com. Subsequently, the developers of Aport demonstrated extreme sluggishness in the development of their project, setting up a search for a very long time, first on 4 servers, then on 6. Aport learned to index the entire Runet only in November 1997, at which time it was officially launched. By this time, another search engine called Rambler was already successfully operating in the Russian-language segment.

Despite all these circumstances, Aportu until the early 2000s. managed to successfully compete with the main market players Rambler and Yandex, and be included in the list of search leaders in Runet. Subsequently, the company that created this search system was bought out by a telecommunications holding, all development was stopped, and Aport quickly lost its position, losing ground to its main competitors.

At the moment, Aport is an electronic trading platform with a large database of firms and companies offering more than 8 million items of goods in 1,400 categories.

Rambler

The team at the telecommunications company Stack decided to create an original Russian search engine back in 1994. By that time, Stack already had some experience in working with the Internet, servers and websites. Working with the Russian segment of the Internet, the company’s specialists determined that foreign search engines practically do not perceive the Cyrillic alphabet and pages with multiple encodings, and index Runet sites very poorly.

Rambler" translated from in English- “wanderer”, “tramp”, “loitering person”.

The core of the new search engine was written by programmer Dmitry Kryukov in just a few months. Work on the new machine was financed by the Stack company, whose creator Sergei Lysakov actively helped Kryukov in his very difficult work. The name Rambler and the logo of the future search engine, also Dmitry. The domain rambler.ru was registered on September 26, 1996, and on October 8, a search engine called Rambler was posted by its creator on the network. At that time, the new search engine had indexed 100 thousand documents, which was a thoughtful and strategically important step that allowed Rambler to become the undisputed search leader on the RuNet for several years.