Maxim Polyakov: what search engines are made of and how Google became the main one (and still is)

The greater the amount of information on the Web, the more we need search engines. The first place among them has long been firmly occupied by Google. Why its appearance has revolutionized and what to expect from the development of web tools in the future, said MGIMO professor and media manager Maxim Polyakov. Lifehacker made a summary of his lecture at an interdisciplinary conference Screenshot_2023, which took place in Moscow on March 25 and 26.

When and why users needed search engines

The volume of information on planet Earth is constantly increasing. For centuries, data has been accumulated on analog devices - first in manuscripts and books, later - on vinyl records, on film and video cassettes.

In 1993 there were digital data storage. And since then, the share of digital information has been growing. In 2002, a turning point came: the volume of data in digital became equal to analog sources. The digital age has begun. Since then, the stock of digital data has grown exponentially, and with it, the total amount of information on the planet has also increased.

A problem arises: it is not clear how to quickly find what you need in this huge amount of information. That's what search engines do.

In 1998, when Google first entered the market, there were several search services on the Internet. For example, AltaVista and Lycos portals were available in Russia, and in total there were more than a dozen such resources: Netscape, Ask Jeeves, AOL, MSN and others. Google became one of them and defeated competitors.

Conclusions about how it is better than other search engines, back in 1999, were made by the Spanish scientist Ramon Salvera. In the late 90s, he came across an interesting dissertation from Stanford University. It was called "Research papers about Google and the Web Base" and was published in 1997. Its authors were graduate students Sergey Brin and Larry Page.

Salvera read the dissertation and tested the search string. And he immediately wrote an article for the city newspaper "Navarre Diary", which he called "The Google Revolution".

How search engines worked before Google

Enter in the search bar request - for example, "hardware store". The algorithm finds documents that contain this phrase. Next, the search engine counts how many times the keywords are repeated on the page. Those documents, where there are more such references, fall on the first lines of the issue.

Important: the search engine does not know what kind of pages these are - maybe store sites, or maybe just resources that write about construction. Therefore, the algorithm cannot determine how useful documents in the search results will be useful to the user. As a result, a person receives a long list of sites and he has to open each page and search for the right ones among them for a long time.

How Google fundamentally changed the rules of search

The main thing that appeared in Google is PageRank, a new search algorithm. Brin and Page decided that it would not be worthwhile to search the Internet for individual documents in turn on request, and then issue them in a list. You need to explore them all at once. At the same time, it is important to select from the general list those that will be the most significant for the user.

The most difficult thing is to find the right criterion that would show the authority of the document. To do this, the authors have developed a special formula and called the resulting value of PR - PageRank.

The revolutionary discovery of the creators of Google - the choice of criteria for assessing the importance of the document. They became links.

The main thing that was taken into account in the new algorithm is how many times other resources refer to the document. The one that is mentioned most often becomes the first in the ranking and is recognized as the most authoritative. For example, if seven out of ten sites refer to one, then the leader is he.

As a result, several nodes are selected that have the most links. Further, the formula takes into account whether our found “number one” mentions any documents. If there are such pages, they immediately rise in the ranking. Document links from the top of the list always weigh more than links from sources below.

Eventually user sees sites in the search results that are really useful for him. He does not need to spend time analyzing pages where keywords are mentioned - Google has already done this for him.

How Google Came to the Top and Why It Holds the Lead

The founders of Google created not only the search algorithm. They developed the architecture of the company and came up with the main principles of its work.

Brin and Page registered Google in 1998 and started building their future. corporation in the garage of Susan, Brin's friend. On the whiteboard, the guys wrote: “Focus on the user.” A little later, they formulated the mission of the company.

For Google, the priority from the very beginning was the one who sets the search query, and not the organizations that created the sites and are waiting for new customers. Therefore, the company immediately decided: advertising should be clearly separated from search results, and a person should not confuse organic search results with paid ads. This rule attracted users and helped to increase the audience.

The company quickly received the first investments. Google entered the top ten most popular Internet resources in early 2001 and easily rose to the fifth line. In 2004, the project ceased to be a "garage" one and went public. In 2006, the corporation bought YouTube and the company that developed the Android operating system. These acquisitions accelerated growth.

In March 2006, Google rose to first place among all Internet resources. In November 2009, he briefly lost the championship to Yahoo!, but in January 2010 he regained his position and still remains in the lead.

Today, Google for users is several platforms:

• Search algorithm.
• Service for contextual advertising.
• Browser.
• Operating system for smartphones.
• Dozens of services for ordinary users and businesses: mail, maps, documents, spreadsheets and much more.

An important reason for the company's success is corporate culture.

Brin and Page, when choosing a new employee, proposed to conduct a thought experiment: imagine that you have to spend six hours with this person at the airport, because flight was delayed. How comfortable will he be?

In this way, they created an environment in which people fit together. It was necessary for the development of the company and the emergence of innovations.

All this has led to the fact that today among all user requests to search engines, Google accounts for 92.21%. This is a global monopoly.

What is good about "Yandex" - the main competitor of Google in Russia

There are only two countries in which Google has serious competitors, and the company can't do anything about them. This is China - 68.66% of searches there are on the local Baidu portal. And Russia - we have more than 45% of requests received by "Yandex".

This search engine was created by Ilya Segalovich and Arkady Volozh in 1997 - that is, almost simultaneously with Google. Today Yandex is the No. 1 site in Russia.

Like Google, it's a platform system. The platform combines several different services that cover the different needs of the client. In the first place here is also a search engine algorithm. It is very good, but not as much is known about how it works as about the Google algorithm.

In addition to the search engine, Yandex has an advertising platform and browser. Plus additional services: mail, maps, disk, market and others. The greatest profit today is brought by Yandex Taxi and delivery services.

What changes ChatGPT in our relationship with search engines

In 2023, that is, right now, another revolution is taking place. Google is still the leader, and for some time it will hold on to the first place, but we do not know what will happen next.

In February, CEO Microsoft Satya Nadella, during a speech at his company's office, announced that a new era had begun. According to him, the entire history of web technologies is divided into three stages:

• Personal computer + server.
• Mobile technologies + cloud storage systems.
• Artificial intelligence.

We are at the very beginning of a new, third era. Even the start date is known - Satya Nadella said this on February 8, 2023.

ChatGPT gained a million users in 5 days. This is the fastest rise in the history of the Internet. The previous leader, Instagram*, took 75 days to achieve the same result.

The neural network is now integrated into the Bing search engine. The question in the search bar sounds like this: "Ask me anything", or "Ask me something." ChatGPT answers complex questions, and does it in detail. Moreover, it is able to cope with tasks that were previously used in Turing test. Before that, he always helped to distinguish a person from a machine algorithm, but now everything has changed.

Sergey Brin returned to Google after five years of vacation. He had to go back to work because the emergence of neural networks in the company declared a red alert level. Google is trying to make active bard — own chatbot. It is believed that its parameters are five times greater than the capabilities of GPT. But this neural network has not yet been fully operational.

*Meta Platforms Inc. activities. and its social networks Facebook and Instagram are prohibited in the territory of the Russian Federation.