The Publishing project

I grew up with search. From the early days of Yahoo—built as a directory of links—to the modern algorithmic engines we have spent our careers optimizing for, search has been our compass. Today, we are witnessing a rapid transition from this classic “index-and-referral” model to an “all-in-one answer engine.” That shift is not merely a software update; it is a structural realignment of the web’s economic and cultural foundation.

The web is reaching a critical inflection point. Generative AI is being embedded directly into browser interfaces and search products, changing not only how people find information but also how websites earn attention, trust, and revenue. As platforms move from link directories to conversational assistants, developers must rethink how they build, publish, and sustain work on the open web ¹, ², ³.

To understand the magnitude of this change, we need to look closely at how search engines are dismantling the model that made the web valuable in the first place—and at how developers are responding by looking to the past to rebuild something more resilient.

From Directories to Synthesis: The Evolution of Search

To understand why the current transition feels so disruptive, we have to look at the architectural history of search. The web’s discovery systems have evolved through four distinct technical eras, each redefining the relationship between content creators and platforms.

1. The Directory Era (Manual Curation): In the early 1990s, the web was organized manually. Platforms like Yahoo, DMOZ (the Open Directory Project), LookSmart, and EINet Galaxy operated as hierarchical, human-curated directories of links. If you built a website, you submitted it to a category, and editors reviewed and cataloged it. The search platform was explicitly a directory pointing outward.
2. The Algorithmic Era (Link Graphs & PageRank): As the web scaled exponentially, manual curation became impossible. Google revolutionized discovery by treating the web as a mathematical graph. PageRank utilized inbound hyperlinks as citations, valuing pages based on the quantity and quality of links pointing to them. Alongside other automated crawler-based engines like AltaVista, Excite, and HotBot, search engines during this era acted as highly efficient navigational pipelines. They were designed to index the web and route users off their search page and onto target sites as quickly as possible.
3. The Semantic Era (Entities & Knowledge Graphs): In the 2010s, search evolved from matching string keywords to understanding things (“entities”). Google introduced its Knowledge Graph, and Microsoft deployed Bing Satori, allowing search engines to understand the relationships between people, places, and facts. Specialized computational engines like WolframAlpha focused on answering objective facts algorithmically. This era saw the introduction of “Rich Snippets” and “Featured Snippets”, the first step toward answering queries directly on the search page, signaling the initial friction in the classic referral model.
4. The Synthesized Era (RAG & Vector Embeddings): Today, we are entering the synthesized era. Using Retrieval-Augmented Generation (RAG), engines like Perplexity AI, Google’s Gemini-powered AI Overviews, OpenAI’s SearchGPT, and developer-centric systems like Phind do not merely search an index for relevant links. They scrape the underlying text, vectorize the data, project it into high-dimensional embedding spaces, and synthesize a single, cohesive answer using a Large Language Model (LLM). The search engine is no longer a router; it is the final destination.

The Broken Social Contract of Search

For decades, the web operated on a mutually beneficial social contract between website owners and search engines: creators allowed search algorithms to crawl, index, and scrape their content, and in exchange, search engines sent highly targeted, intent-driven traffic back to those websites ⁴. That traffic fueled discoverability, built audiences, and supported monetization through ads, subscriptions, and products.

Recent shifts toward AI-generated summaries directly disrupt this historic exchange ¹. Companies like Google have moved beyond simple link routing and now use Gemini-powered systems to answer questions inside search itself, often allowing users to engage in a full AI-mediated discussion without ever visiting a third-party website. This new paradigm prioritizes immediate, synthesized answers inside the search interface over directing users to external links. When a search engine functions primarily as a closed chatbot, the incentive to surface external URLs declines drastically ², ³. Studies and real-world analytics show that “zero-click searches”—queries that end on the search page without a single click to an external site—are skyrocketing, sometimes accounting for 60% to 70% of search traffic in informational niches.

From an engineering perspective, this is a crisis of extraction. LLMs use retrieval-augmented generation to scrape, vectorize, and synthesize content on the fly, often consuming creators’ work while redirecting the value away from the sites that produced it. That dismantling of the link-based search ecosystem breaks the core incentive structures that keep independent creators publishing ⁵. Writing high-quality, deeply researched, and technically accurate content requires immense time, effort, and capital. If creators receive no referral traffic, the economic model supporting their work collapses. The motivation to produce open, accessible, and high-quality human content diminishes, threatening to leave the open web vacant and starved of fresh data—ironically starving the very AI models that rely on this content for training ¹.

To visualize how these automated answers are altering user behavior and putting independent websites in jeopardy, watch Kevin Powell’s “Google might have just killed websites”. He breaks down the technical and economic implications of Google’s pivot away from outward referrals:

The Crawling Crisis: To Block or Not to Block?

Because this social contract has collapsed, developers are facing an unprecedented architectural and ethical question: should we continue to allow AI engines to crawl our websites?

Historically, developers managed crawlers using robots.txt, a voluntary, non-enforceable standard established in 1994. Under the old contract, “politeness” was a mutual agreement. Today, that agreement is fundamentally broken. AI crawler traffic now represents a large and growing share of web requests, and poorly written bots like ByteDance’s Bytespider or Meta’s Meta-ExternalAgent frequently hammer application origins, ignoring rate limits, bypassing traditional robots.txt blocks, and triggering expensive database queries.

To navigate this landscape, modern developers must move away from blanket bans and adopt a purpose-based scraping control strategy. We need to separate AI bots into two distinct functional categories:

1. Offline Training Crawlers (The Data Sinks): Bots like GPTBot, ClaudeBot, and Google’s Google-Extended flag exist purely to harvest and ingest your content to train future foundational models offline. Allowing them consumes massive server bandwidth and compute resources for zero immediate return. Fortunately, blocking them does not harm your organic Google search rankings. For example, blocking the Google-Extended token prevents Gemini from training on your text, while leaving the primary Googlebot completely free to index your site for standard search results.
2. Real-Time Retrieval Crawlers (The AI Discovery Layer): Bots like OAI-SearchBot, Claude-SearchBot, and PerplexityBot crawl your site dynamically to retrieve context for active user queries. If you block these live retrieval bots, you vanish completely from AI-powered search results, losing highly qualified referral traffic that is growing rapidly year-over-year.

The Real Cost of Politeness

Remaining entirely open to AI crawlers is no longer a free choice. For any site running dynamic application logic (rather than static HTML), aggressive AI bots create immediate infrastructure costs. When a bot loops through parameter-heavy URLs, search filters, and checkout paths, it triggers expensive application CPU cycles.

Projects like Read the Docs reported cutting their global bandwidth usage by 75% simply by blocking aggressive AI training crawlers—saving thousands of dollars in monthly cloud hosting bills.

Instead of relying on the honor system of a robots.txt file, developers are increasingly moving bot management to the network edge. Using Edge Web Application Firewalls (WAFs) like Cloudflare or Fastly, developers can write granular rules that block resource-heavy offline training crawlers at the network edge while preserving paths for real-time, traffic-generating AI retrieval agents.

Re-engineering the Economic Model

With search traffic dwindling, the downstream effect is a collapse of traditional creator revenue streams. The classic model, driving mass, anonymous traffic from Google to a website optimized for banner ads or basic affiliate links—is no longer sustainable. As AI answers user queries directly, click-through rates plummet, taking ad impressions and affiliate conversions down with them.

To survive, independent creators and programmatic ad networks are fundamentally transforming how web traffic is monetized.

The Shift in Ad-Tech Paradigms

The transition to AI-centric search also upends the digital advertising business ⁶. Traditionally, platforms placed ads alongside search results or directly on content pages. If users no longer click through to websites, traditional banner ad and pay-per-click models lose their effectiveness ¹, ⁵.

As the volume of traditional page views decreases, legacy advertising networks are being forced to evolve:

• From Page Banners to Conversational Agents: Major search providers like Google are experimenting with tokenized bidding and integrating advertisements directly into AI conversations ¹, ⁶. Instead of a static image, an advertiser provides a creative brief and targeting goals, and the AI dynamically generates an ad to match the specific conversation. For networks like AdSense, this could mean shifting from placing ads on publishers’ websites to developing systems where creators receive micro-transactions when an AI utilizes their specific content or expertise to generate a conversational ad.
• Premium Context and Native Integrations: While generic traffic networks may suffer, specialized ad networks (like Carbon Ads, which targets developers and designers) might actually thrive by leaning into human curation. If general search traffic declines, advertisers will pay a premium to reach guaranteed human audiences. We will likely see these networks double down on native placements within high-quality newsletters, exclusive communities, or deeply integrated sponsorships within specific developer tools (e.g., suggesting a sponsored cloud service directly within an AI coding assistant).
• High-Efficiency Ad Delivery: With fewer overall visitors, publishers must extract maximum value from the traffic they do receive. Expect a push for extreme technical performance—such as utilizing bfcache (Back-Forward Cache) and the Speculation Rules API (to pre-render pages in the background)—to ensure that the moment a user does land on a page, the ads load instantaneously and render flawlessly, guaranteeing that every rare impression is counted.

The Video Moat: Developer Content as Performance

While text-based blogs and documentation face the immediate threat of AI summarization, many developers are shifting their publishing strategies to video platforms. Video acts as a distinct “moat” against AI replacement. For developers, this represents a shift from static text to code as performance.

Human beings inherently crave visual proof, real-time problem solving, and human validation—elements that a text-based AI chatbot cannot easily replicate. In an era of AI “slop,” watching a developer spin up a terminal, write a script, and debug a runtime error in real-time provides a level of authenticity that an LLM-synthesized article cannot match.

However, the technical and discovery dynamics of this visual shift vary wildly depending on the platform’s infrastructure:

• The YouTube / Google Dynamic (Transcripts & Timestamps): Because Google owns YouTube, video content is deeply integrated into modern search architectures. To feed its answer engine, Google’s Gemini-powered search often scrapes video transcripts and metadata on the fly. Instead of routing users to watch a video, AI search now risks creating a “zero-click video” scenario. If a developer searches, “How do I configure this specific Docker container?” the AI can extract the exact lines of code from a video transcript, summarize the steps, and present them in a text box. To survive this, developer-educators are pivoting away from simple “syntax tutorials” (which are easily summarized) and leaning into complex architectural design, opinionated code reviews, live-programming challenges, and narrative storytelling where the value lies in watching the thought process rather than copying the final script.
• Independent and Premium Hosting (The Self-Hosted and SaaS Alternatives): Because hosting high-bandwidth video is historically expensive, developers looking to bypass corporate ad-tech algorithms are leveraging alternative models. Platforms like Vimeo or Wistia offer creator networks that monetize through subscription software rather than an attention-ad algorithm. For developers selling advanced engineering courses, embedding videos hosted on independent platforms within private, token-gated developer portals acts as a total shield against search-crawler scraping. Similarly, self-hosted video & peer-to-peer networks like PeerTube (driven by ActivityPub) allow developers to publish video content on decentralized instances, using WebTorrent-based peer-to-peer distribution to offload server bandwidth and keep content out of corporate indexes.

Relational, Subscription-Based Models

Beyond ads, creators are shifting away from transactional, traffic-based monetization toward relational, subscription-based structures. This approach focuses on extracting deeper value from a smaller, highly engaged core audience rather than relying on sheer volume—a modern application of Kevin Kelly’s famous “1,000 True Fans” theory.

• Direct Subscriptions and Memberships: Platforms like Substack, Patreon, and Ghost, alongside native website paywalls integrated with Stripe webhooks and JWT-based authorization gates, allow creators to offer exclusive, premium content to their most dedicated followers for a recurring monthly or annual fee.
• Paid Communities: Moving beyond one-way content delivery, creators are building thriving, gated communities. By charging a monthly fee for access to private Discord servers, Slack teams, or specialized forum software, creators monetize peer-to-peer networking, accountability, and direct access.
• Consulting, Services, and Digital Products: Instead of monetizing the attention of an audience via programmatic ads, creators are increasingly monetizing their expertise. This includes offering one-on-one consulting, cohort-based online courses, downloadable toolkits, or custom templates. A smaller, high-trust audience arriving via an email newsletter or a trusted referral link is much more likely to convert on a high-value product or service than a random visitor arriving from a search engine.

A Return to the “Boring” Internet

As the highly commercialized “veneer” of the internet—characterized by SEO-optimized content farms, intrusive ad networks, and algorithmic feedback loops designed to maximize anonymous page views—begins to fracture under the weight of AI-generated summaries, many developers are advocating for a return to the “boring” web ⁷, ⁸.

This commercialized layer is failing because AI summaries bypass the need to visit ad-laden, keyword-stuffed pages entirely ⁸. For years, much of the web was written for search bots rather than human beings, resulting in articles padded with bloated introductory paragraphs to satisfy search engine optimization (SEO) length requirements. By automating the extraction of those key points, generative AI is rendering this algorithmic style of writing less useful.

This realization is driving a massive resurgence of the “indie web,” a movement that relies on foundational, open internet protocols rather than proprietary, corporate platforms. Unlike platforms or services that seek to lock users into walled gardens to monetize their attention, open protocols empower users to retain complete ownership of their content, data, and social connections ⁷, ⁸.

Historically, this return is a full-circle evolution. The early web was defined by open protocols, but the commercial floodgates opened wide in 1991 when the National Science Foundation (NSF) eased its Acceptable Use Policy (AUP) on the NSFNET backbone, allowing commercial traffic for the first time. The subsequent decommissioning and complete privatization of the backbone in 1995 fully shifted the web from a public, protocol-driven utility to a commercial playground. What we think of today as the “indie web” is actually a re-engagement with those pre-1991 foundational standards, seeking to escape the monetization-first structures that followed privatization.

This historical shift highlights why modern developers are nostalgic for simpler web patterns. In “The future of the web might lie in its past”, Kevin Powell details the friction between proprietary platforms and open standards, exploring how the indie web is staging a quiet renaissance:

Key protocols driving this movement include ⁷:

• RSS (Really Simple Syndication): An open XML-based format that allows users to subscribe directly to their favorite websites and receive chronological updates in a central reader. It completely bypasses algorithmic curation, ensuring that if a creator publishes an article, every subscriber receives it.
• Email (SMTP/IMAP): Remains a resilient, decentralized protocol that no single tech giant controls. Because anyone can set up an email server or export their subscriber list, newsletters have become a vital tool for independent publishing.
• Federated Networks (ActivityPub): A W3C protocol that connects independent servers into a shared social network (the Fediverse) using JSON-LD payloads over HTTP. It allows users on Mastodon, Pixelfed, or WriteFreely to follow and interact with one another seamlessly, regardless of which server host they choose.

Alternatives for Discoverability and Reach

With Google and other search engines prioritizing internal zero-click answers, relying on broad algorithmic search traffic is no longer a viable primary strategy. To get content in front of target audiences, creators are shifting to alternative discovery methods that bypass traditional search bottlenecks ⁷.

• Doubling Down on Owned Distribution: The most robust defense against search algorithm volatility is establishing a direct, unmediated line of communication with your audience. * Email Newsletters: Because email is built on a decentralized protocol, no single platform can restrict your access to your subscribers. If a service provider changes its terms, you can export your .csv list of subscribers and migrate to another platform. It remains one of the highest-converting ways to distribute content directly ⁷. * RSS Feeds: Providing an RSS feed allows power users and loyal readers to pull your content directly into their personalized aggregators (like Feedly, NetNewsWire, or Inoreader) without relying on social media algorithms ⁷.

• Human Curation and Niche Indexes: As AI search fills with homogenized summaries and web searches become saturated with low-quality, AI-generated “slop,” human curation is becoming highly valued. Niche industry newsletters (such as CSS Weekly, Sidebar, or specialized tech digests) curate the best links from around the web manually. Pitching content to these human curators or building relationships with industry aggregators can drive highly targeted, high-intent traffic ⁷.

• Community-Led Growth and Dark Social: Instead of trying to capture “strangers” via generalized search, creators are going to where communities already gather. Sharing expertise and content within specialized Discord servers, Slack communities, niche forums, and Reddit subreddits provides direct access to a highly relevant audience.

    Furthermore, a massive portion of modern sharing happens via "dark social." The term, originally coined by Alexis C. Madrigal in 2012, describes social sharing that occurs outside the view of traditional web analytics tools. Technically, this happens when users share links through private channels—such as WhatsApp, Signal, iMessage, and Slack—that strip out standard HTTP Referer headers. When a user clicks a link inside these applications, the request arrives at the web server without referral metadata, causing analytics suites to misclassify this highly targeted, high-intent peer-to-peer traffic as direct ("Direct/None") visits. Understanding and capturing dark social signals is vital, as a link shared privately by a trusted colleague converts at an exponentially higher rate than a generic search hit.
  

• The Federated Social Web: The rise of the Fediverse (such as Mastodon) offers a social media discovery engine built on open protocols rather than corporate algorithms ⁷. Because these platforms are chronological and community-driven, quality content spreads through genuine human interaction and sharing, rather than algorithmic boosting designed to trigger outrage or maximize dwell time.

Peer-to-Peer Recommendations (The Retro Web)

Old-school web discovery mechanics are returning as modern web users increasingly crave human-vetted content.

• Webrings: Groups of related sites that link to one another in a circular loop (usually via “Previous” and “Next” buttons at the bottom of the page). Webrings allow creators in niche circles (like indie game developers, designers, or personal bloggers) to share their collective audiences.
• Blogrolls: Curated lists of recommended links placed on a creator’s sidebar. If a trusted creator recommends your site on their blogroll, their audience is highly likely to click through, creating a decentralized web of trust ⁷.

Conclusion: Navigating Uncertainty

The internet is undergoing a dual transformation: the methods for building the web and the habits for consuming it are changing simultaneously. The decline of ad-heavy, SEO-optimized content farms might improve the user experience, but losing dedicated independent creators would drastically reduce the web’s value.

The exact shape of the internet’s next era remains unclear. However, developers, content creators, and audiences must adapt by embracing open protocols, prioritizing direct connections, and finding new ways to intentionally discover and appreciate the creators who provide genuine value ⁷, ⁹.