From SEO to AI Engine Optimization

Introduction

The ideas in this article come from something I noticed while looking at my blog’s data recently. Starting in April, I saw that many AI search engines had popped up in the traffic sources in my blog’s backend. At first, I didn’t think much of it, just treating it as a kind of recommendation channel similar to niche websites. But by June, things had changed. I found that visits from AI sites like Perplexity, Kimi, and Coze had actually overtaken Baidu search. That’s when I started to take this trend seriously.

My blog mainly covers AI and software topics, and all the English and Chinese content gets polished and checked by AI. Since I work in Google search engine optimization, I weave a lot of SEO techniques into my content when I write. So, my articles are generally pretty search engine friendly.

But these factors alone don’t fully explain why so many readers are coming to my site through AI channels. This got me thinking, and made me want to take a closer look at the new role AI search engines are playing in how information spreads.

Spotting Industry Trends from Subtle Changes

Since Bing introduced its AI-enhanced search features, the debate about whether AI search will replace traditional search has been heating up in the SEO industry. This discussion isn’t baseless – from StatCounter’s global statistics, we can clearly see this trend gradually emerging in the US desktop search market.

Specifically, Google’s search share has been showing a slow but noticeable downward trend since October 2023. The data reveals that Google’s market share has gradually decreased from nearly 80% in mid-2023 to around 73% in June 2024. This seemingly small change actually reflects a significant shift in the search market landscape.

At the same time, Bing’s market share has been steadily growing. From October 2023 to June 2024, Bing’s share increased from about 11% to around 13%. While the increase isn’t huge, this consistent growth trend is worth noting and is likely directly related to its newly introduced AI-enhanced features.

It’s worth noting that the chart lacks direct data on other AI search tools. However, I can infer that emerging AI search tools likely absorbed some of Google’s lost market share. This suggests a shift in user search behavior.

The Rise of AI Search: A New Era Arriving?

With Perplexity’s new round of funding and OpenAI’s latest launch of Search GPT, we can easily foresee a future. AI search will likely disrupt certain markets in some aspects. Or it may occupy part of the space currently held by traditional search engines. This trend prompts us to think deeply about the future of the search industry.

Timeline

November 30, 2022: OpenAI releases ChatGPT

• Laid foundation for AI-driven search with advanced Q&A capabilities.

February 7, 2023: Microsoft launches new Bing

• Integrated ChatGPT, marking the start of AI-enhanced search era.

March 2023: Perplexity AI completes initial funding

• Raised $25.5M at $210M valuation, showing investor confidence.

June 2023: Anthropic launches Claude 2.0

• Enhanced long-text processing, opening new possibilities for AI-assisted information retrieval.

December 2023: Perplexity AI completes Series B

• Raised $73.7M at $520M valuation, proving AI search market potential.

April 23, 2024: Perplexity AI becomes unicorn with Series C

• Raised $62.7M at $1B valuation, solidifying position in AI search market.

July 25, 2024: OpenAI officially launches Search GPT

• Strengthens position in search, signaling potential major shifts in the landscape.

Will Traditional Search Engines be Replaced by Generative AI?

In the past, people mainly searched using keywords. AI search, however, can understand user intent, helping users find information more accurately. This shift could profoundly shape the future of the search industry.

Many habits of modern internet users have actually been shaped by search engines. Most people use search engines to get information, but traditional search engines can’t truly understand the intent behind human queries. As a result, users often simplify complex needs into a few keywords. For example, “I want to buy a 150ml capacity thermos, what’s a reasonable price?” gets boiled down to “cup, price, 150ml”.

This simplification, while adapting to how traditional search engines work, also limits users’ ability to express more complex and precise needs.

Past Google ad strategies and SEO were based on keywords and rankings. For instance, how many pages of results a keyword had, with the top 3 bringing in lots of traffic. This keyword model has also influenced modern social media design, like tag systems.

This deeply ingrained keyword mindset has not only shaped user behavior but also influenced the entire digital marketing and content creation ecosystem:

1. Content creation: Authors tend to use high-volume keywords to increase content exposure, sometimes at the expense of quality.
2. Website structure: Site architecture and navigation often revolve around keywords rather than actual user needs or logical flow.
3. Ad placement: The rise of platforms like Google AdWords stems directly from the keyword bidding model, greatly influencing companies’ marketing strategies and budget allocation.
4. User behavior analysis: Big data analysis often focuses on keyword trends and click behavior, overlooking users’ deeper needs and intentions.
5. Social media algorithms: Content recommendation algorithms on social platforms largely rely on keywords and tags, sometimes leading to information bubbles.

The emergence of AI search might change this situation:

1. Natural language queries: Users can express their needs in complete sentences or even paragraphs, no longer needing to deliberately choose keywords.
2. Intent recognition: AI can understand the real intention behind queries, like distinguishing whether “cup price” means wanting to buy a cup or doing market research.
3. Context understanding: AI can consider factors like user’s search history, location, and time to provide more personalized results.
4. Multi-turn dialogue: Users can gradually clarify and refine their needs through multiple interactions, rather than inputting all information at once.

As a search engine optimization professional, I can’t help but wonder: Past SEO and SEM industries were built on search engines – you could say without search engines, there’d be no SEO. So, following this trend, will we see a new kind of optimization strategy geared towards large language models in the future?

SEO vs GEO: A Paradigm Shift in Content Optimization

What is SEO (Search Engine Optimization)?

Search Engine Optimization (SEO) encompasses strategies and techniques aimed at improving a website’s ranking on search engine results pages (SERPs). Its fundamental principle involves aligning websites with search engine algorithms’ preferences to achieve higher rankings and increased organic traffic.

Basic Principles and Importance of SEO

SEO’s significance is clear. In our digital era, search engines serve as the primary information gateway for most people. Higher rankings translate to greater exposure, leading to increased traffic, potential customers, and revenue. At its core, SEO is a guessing game—anticipating search engine algorithms’ preferences and tailoring content accordingly.

Introduction to Traditional SEO Strategies

Traditional SEO strategies primarily focus on catering to mainstream search engines like Google. These include:

1. Keyword optimization: Strategically distributing target keywords throughout webpage content.
2. Content structure: Utilizing heading tags (h1, h2, h3) to organize content effectively.
3. Meta tag optimization: Refining title and description tags.
4. Link building: Acquiring high-quality external links to boost website authority.
5. Technical optimization: Enhancing website loading speed and ensuring mobile-friendliness.
6. Content creation: Developing “SEO-friendly” content, such as FAQ sections and long-form articles.

These strategies essentially cater to Google’s “black box” algorithm. SEO practitioners often find themselves in a position similar to courtiers trying to decipher a ruler’s wishes, constantly probing and devising new tactics. The market is saturated with SEO tutorials teaching various techniques, from building perfect title hierarchies to optimizing anchor text and incorporating Q&A sections. While these methods have proven effective in the past, they’ve led to a somewhat distorted approach to content creation.

From SEO to GEO: The Need for a New Optimization Method

Generative AI’s Impact on Information Retrieval

Generative AI is revolutionizing information access. Unlike traditional search engines that provide a list of links, generative AI offers direct answers, comprehensive information, and even engages in multi-round conversations. This shift eliminates the need for users to navigate multiple web pages, challenging the very foundation of traditional SEO.

Limitations of Traditional SEO Strategies in the AI Era

1. Keyword stuffing loses effectiveness: Generative AI prioritizes semantic understanding over simple keyword matching.
2. Diminished link importance: AI may value content quality more than the quantity of external links.
3. Technical optimization alone is insufficient: While website performance remains important, content substance and utility become crucial.
4. “SEO-friendly” content may miss the mark: Material optimized for search engines might not meet AI’s requirements.

The core issue with traditional SEO strategies lies in their attempt to algorithmically define “good content” and “user-helpful content.” Algorithms are inherently rigid and often opaque, leading to distorted practices where content creators focus on pleasing algorithms rather than truly serving users.

This situation characterizes the current SEO landscape—an ecosystem built on speculation, techniques, and ever-shifting “best practices.” Despite good intentions, search engines’ algorithmic approach to defining and rewarding “good content” has paradoxically led to declining content quality and user experience.

These concerns have long been discussed in the SEO community. A significant shift occurred in March 2024 when Google’s algorithm update and stricter policies on AI content caused widespread ranking drops and keyword losses for many websites. This update not only challenged traditional SEO strategies but also put AI-reliant content creators in a difficult position.

Amidst this industry upheaval, a paper titled “GEO: Generative Engine Optimization” caught my attention. Its timely appearance offered a fresh perspective and potential solutions to the SEO industry’s current challenges.

GEO: A New Approach to Optimization for Large Language Models

Thirty years ago, the invention of traditional search engines fundamentally changed how information was accessed and disseminated globally (Brin and Page, 1998). While these search engines are powerful and have spawned numerous applications including academic research and e-commerce, they are limited to providing users with lists of relevant websites for their queries. Recent successes in large language models have paved the way for superior systems like BingChat, Google’s SGE, and perplexity.ai, which combine traditional search engines with generative models. We call these systems Generative Engines (GEs) because they search for information using multiple sources and generate multimodal responses.

From a technical perspective, generative engines retrieve relevant documents from databases (such as the internet) and use large neural network models to generate responses based on these sources, ensuring attribution and providing users with ways to verify information.

GE vs SE: Key Differences and Innovations

Content Presentation

Traditional search engines present a linear list of websites, requiring users to click these links to access relevant sites for information. In this model, a webpage’s average ranking in search results is a good indicator of visibility. This simple ranking metric, however, doesn’t apply to generative engines.

Generative engines provide rich, structured responses that directly embed website content into answers as inline references. These embedded contents may vary in length, position, and style. Thus, in generative engines, the concept of content visibility becomes more nuanced and multifaceted.

Algorithmic Differences

Google’s search algorithm uses hundreds of factors, but some key principles include PageRank, content relevance, RankBrain, user experience signals, and freshness. PageRank evaluates the quality and quantity of links pointing to a webpage to determine its importance. Content relevance analyzes webpage content to determine its relevance to search queries. RankBrain uses machine learning to understand the context of searches and content. User experience signals consider factors such as page load speed, mobile-friendliness, and user engagement. Freshness often prioritizes recently updated content for certain queries.

Large language models like GPT work through several key mechanisms. First is the attention mechanism, which allows the model to focus on different parts of the input text when generating output. Second is contextual understanding, meaning the model can understand and generate content based on broader context, not just simple keyword matching. Models also utilize transfer learning, meaning they are pre-trained on vast amounts of text, allowing them to leverage general knowledge in specific tasks. Additionally, models process text by breaking it down into token sequences to understand relationships between words and phrases. Finally, models have generative capabilities, able to create coherent and contextually relevant text, rather than just selecting and combining existing content.

What is Generative Engine Optimization?

Similar to traditional search engine optimization, this optimization method can help content creators increase the visibility of their content in generative search engine responses. This concept stems from the idea of traditional Search Engine Optimization (SEO) but is targeted at the new generation of search engines centered on large language models.

GEO Practice: How to Optimize Your Content for AI

The emergence of generative engines has not only changed how information is presented but also profoundly impacted the entire internet ecosystem. For users and developers, this is a significant advancement: users can access information more quickly and accurately, while developers can provide more precise, personalized responses, thus improving user satisfaction and potential revenue.

However, this progress brings unprecedented challenges to third-party stakeholders – websites and content creators. In the traditional search engine model, users needed to click through to websites to obtain information. Generative engines, on the other hand, provide comprehensive answers directly on the search results page, potentially leading to a significant decrease in website traffic. According to Maayan (2023), this change could seriously affect many small businesses and individual creators who rely on online traffic and visibility for their livelihood.

More problematic is the opacity of generative engines. They operate like “black boxes,” with internal mechanisms invisible to the outside world, making it difficult for content creators to understand and control how their content is processed and displayed by these systems. This uncertainty brings new challenges to content optimization.

Faced with this situation, content creators need to rethink their strategies. Traditional SEO techniques may no longer be effective. Instead, there’s a need to understand how AI interprets and uses content, and how to make content more influential in generative responses. This is why we need a new optimization method – Generative Engine Optimization (GEO) – to address this new AI-driven era.

Core Strategies of GEO: Theories and Hypotheses

In discussing the core strategies of Generative Engine Optimization (GEO), we must emphasize that these strategies are mostly theories and hypotheses based on understanding the characteristics of large language models. They have not yet been extensively tested in practice, and their effectiveness still needs further validation. Here are some widely discussed potential strategies:

1. Content Structuring and Semantic Optimization: Theoretically, clear content structure and rich semantic information may help AI systems better understand and extract information. This might include using appropriate heading hierarchies, lists, and tables to organize information.
2. AI-Friendly Metadata and Schema Markup: Based on AI’s ability to process structured data, using standards like schema.org to mark up content might increase the likelihood of content being correctly understood and used. However, the actual effect of this method still requires more research.
3. Conversational Content Optimization: Given the conversational nature of generative AI, creating content structures that simulate natural dialogue might increase the chances of content being selected. This includes anticipating and answering questions users might ask.
4. Multimodal Content Strategy: As AI systems’ ability to handle various media types continues to improve, providing multimodal content (such as text, images, videos, etc.) might increase the comprehensiveness and attractiveness of content.

It’s worth noting that the “GEO: Generative Engine Optimization” paper mentioned some strategies, such as adding citations, increasing statistical data, and improving fluency. While these strategies showed some effect in the paper’s experiments, their performance in broader practical applications remains to be seen.

The Essence of Search

What is the essence of search? Search stems from humanity’s fundamental need to acquire knowledge, satisfy curiosity, and solve problems. This is why we see people worldwide engaging in information retrieval on various platforms every day.

The era of search engines is relatively short—only 26 years since Google’s search engine was invented in 1998. Yet, in this brief period, search engines have profoundly altered people’s daily behaviors and marketing methods. A simple example is users’ habit of searching for keyword combinations like “sneakers + price” on Google. The user’s intent might be to buy a cost-effective pair of sneakers, but the same keywords could be used by wholesalers or advertisers for market research. In other words, while keywords simplify the user’s search experience, they can’t fully satisfy diverse search intentions.

AI, however, brings a different approach. Search engines enhanced by large language models can more accurately identify users’ needs and purposes, thus providing better service. As AI search develops, we’re witnessing a massive transformation in the search field. Semantic search is on the rise, enabling search engines to understand the overall semantics and context of queries, not just match keywords. Long-tail keywords and natural language queries are becoming increasingly important, changing the way content is created.

Marketing strategies are also shifting from simple keyword targeting to more complex intent targeting, requiring deeper user research. Websites and applications are evolving to support more natural language interactions and multi-turn dialogues, which could radically change user interface design. Personalized and dynamic content recommendation systems are emerging, based on AI’s understanding of users’ deeper needs.

Therefore, in the post-SEO era, the key is no longer simple keyword placement and expansion, but accurate identification and understanding of user intent. SEO strategies are also undergoing transformation, shifting from traditional keyword optimization to establishing topic authority and meeting user intent. Even social media platforms are evolving, developing more intelligent tagging systems and content discovery mechanisms.

This also explains why both Google and AI-driven search engines favor authentic, thoughtfully human-created content. Such content helps algorithms better identify humanized intentions, thereby providing more precise and valuable search results. In this new era, the authenticity and depth of content will be more important than ever, as they can better reflect and meet users’ actual needs.

References:

Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., & Deshpande, A. (2023). GEO: Generative Engine Optimization. arXiv preprint arXiv:2311.09735. https://arxiv.org/pdf/2311.09735

Foundation Inc. (n.d.). Generative Engine Optimization (GEO): The Future of SEO. Retrieved from https://foundationinc.co/lab/generative-engine-optimization

Nguyen, B. (2023, August 28). Google SGE links to organic search results in 75% of queries, study finds. Search Engine Land. https://searchengineland.com/google-sge-links-organic-search-results-study-436275