The Publishing project

On August 23, Nolan Lawson published The asteroid currently hitting frontend web development.

The article presents a bleak future for frontend development as a profession, with AI primarily responsible for the expected decline in the number of people working in the field.

I have a different perspective. AI is already having a significant effect on frontend development, but its longer-term impact will depend on how it changes the systems around the work, not only how quickly it can generate code.

The immediate impact of AI

There is no doubt that AI is changing frontend development, from automating repetitive tasks to providing code suggestions. These changes also create costs and challenges that are easy to miss when we focus only on productivity.

Two recent examples illustrate this immediate impact, one on the people who publish content and one on the people who consume it.

Unpaid infrastructure costs

Website and account owners are legally and contractually responsible for the bandwidth costs generated by traffic to their sites.

When a user visits a website, requests an API, or downloads a file, the host sends that data from its servers to the user’s browser. This process is known as data egress or outbound data transfer. Cloud providers and web hosts track this traffic and bill the account owner according to the hosting model: metered (pay as you go), fixed allocation with overages (pay for usage beyond a fixed allocation), or unmetered.

These costs contributed to Axel Rauschmayer taking his blog and the online versions of his books offline.

Dear visitor! Two things happened recently:

• The income from my book sales went from being enough for me to live off (2024) to zero (2026).
• The traffic to my blog and my books (which were free to read online) increased beyond what I can currently afford. Virtually all of it comes from AI crawlers, so there is no ad income.

Therefore, I’m taking my blog and my books offline so that I can decide what to do next – especially w.r.t. AI companies stealing my work. This may take a while (think months): I don’t have any energy to deal with this right now.

I apologize for this (hopefully temporary) inconvenience to my human readers! You can still buy my books at Payhip.

This example exposes a larger problem. The web was built on a trust model that is difficult to sustain when automated systems can copy content at scale, while the available protections remain optional and often insufficient.

Addressing these challenges requires several layers: robots.txt rules, firewall rules, rate limiting, and challenges for suspicious traffic, such as CAPTCHA or tools like Cloudflare Turnstile.

For more information, see How to block AI crawlers.

AI as a threat and replacement

The cost of serving content is only one side of the problem. AI can also change what content developers choose to read, buy, or create. Josh Comeau describes this effect in a Bluesky thread, where he considers why developers may be less willing to pay for learning material:

There are likely a lot of reasons for this, but I think the biggest is AI. There’s sort of a double whammy with AI:

1. Many people are wondering whether developer jobs will even exist in a few months, so they’re reluctant to spend time/money learning new dev skills.
2. Even if they do want to learn new dev skills, LLMs can provide personalized tutoring, so there’s less incentive to buy a paid course.

He then makes what I believe is the key statement of the thread:

And for #2, it’s absolutely true that AI can be fantastic learning tool, but it can only answer the questions that you know to ask.

Why the immediate impact matters

These examples are connected. AI crawlers increase the cost of publishing content, while AI assistants can reduce the perceived value of reading or purchasing that content. Together, they can weaken the economic foundation that supports the web’s knowledge-sharing ecosystem.

That is why I am more concerned about the long-term effects of AI on the quality of content, the sustainability of knowledge sharing, and the role of human expertise in the tech ecosystem. The question is not only whether AI will replace some development tasks. It is also whether developers will still have access to trustworthy material that helps them learn what to ask and how to evaluate the answers.

As AI makes individual answers cheaper, the value of human-created content shifts toward selecting trustworthy sources, supplying context, and teaching readers how to evaluate results.

Change the mental model

In my view, the biggest issue with AI today is that many people treat a prompt as a silver bullet that can answer every question and solve every problem. This perception can lead to over-reliance on AI tools and reduce the emphasis on critical thinking and independent problem-solving.

We need a clearer distinction between prompt-based interactions and agentic workflows. Each approach is useful for different kinds of work, and each requires different expectations, skills, and forms of review.

Prompts are not agents

Unless you are familiar with agentic workflows, you likely associate AI with prompts. That is a reasonable assumption for someone encountering AI for the first time, but it is incomplete. A prompt is an instruction in a conversation. An agentic workflow can use multiple steps, tools, and sources of context to pursue a goal. The two approaches have different capabilities and failure modes.

In Why I don’t trust AI prompts, I explain the key differences between prompts and agents, including why prompts alone are often insufficient for complex problem-solving. Relying solely on single-turn prompts can hide the need for broader context, tool use, and human review.

This distinction matters because your mental model affects how you use and evaluate AI. If prompts are your only model for what AI can do, you may expect too much from a single response and overlook the work required to verify it. Better tools will not remove that responsibility; they will make it more important.

Where do we go from here?

The practical response has two parts: limit unwanted access to your content, and limit the influence of unreviewed AI output on your work.

Limit access to your content

The quickest, but incomplete, way to limit AI access to your website’s content is to use robots.txt to control which parts of your site crawlers may access.

Major AI crawler providers generally respect robots.txt when they use their declared user agents, but adherence is voluntary. Smaller scrapers and proxy networks may ignore it.

How robots.txt works

Robots.txt is a voluntary standard that lets website owners specify which parts of a site automated agents should not access. It was originally created for search engines and mirroring spiders.

Search Engine World’s research tested whether 12 AI assistants respect robots.txt. It found that major AI crawlers generally respect the directives when they identify themselves honestly, while many smaller or less reputable crawlers may ignore them.

Compliance is only one way AI crawlers differ. They also differ in what they do with the content they retrieve. Two broad categories are archival crawlers and search crawlers that retrieve content for retrieval-augmented generation (RAG) systems.

• Archival crawlers primarily store web content for future reference. Companies operating AI archival crawlers have been sued over making unauthorized copies of copyrighted material. Publishers have also restricted the Internet Archive’s Wayback Machine because of concerns that AI companies could scrape archived copies of their content for model training and other uses.
• RAG-based search crawlers retrieve relevant, up-to-date information from the web for an AI-powered application or search interface. They are often activated by a user’s request.

The legal risk is not hypothetical. Publishers, authors, music companies, and image libraries have sued AI companies for illegal or unauthorized access to and use of their data and copyrighted work. The New York Times v. Microsoft and OpenAI accuses the companies of using millions of copyrighted news articles to build products that compete with the publication. Authors including Sarah Silverman, Richard Kadrey, and Ta-Nehisi Coates have sued companies such as Meta and OpenAI, while publishers were involved in the proposed $1.5 billion settlement in Bartz v. Anthropic. Music publishers, including BMG Rights Management, have sued Anthropic over its use of protected song lyrics, and Getty Images has sued Stability AI over copyright infringement involving image databases. See this case tracker from BakerHostetler for an overview of AI copyright cases.

As outlined in Genrank’s guide on configuring robots.txt for AI, one way to limit declared AI crawlers is to disallow their user agents in your robots.txt file while explicitly allowing Google and Bing search crawlers.

# Block AI crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Explicitly allow traditional search crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

robots.txt is a request, not an access control. Reputable crawlers such as GPTBot, Google-Extended, and ClaudeBot generally respect its directives, but the file cannot enforce them.

However, not all crawlers are created equal. Malicious bots or less scrupulous AI companies might ignore your robots.txt entirely. For truly sensitive content, don’t rely solely on robots.txt. Use actual access controls like password protection, login requirements, or IP restrictions via your firewall or web server configuration.

For a list of known crawlers, see AI crawlers, search bots, and training agents.

Add stronger controls

To limit access more reliably, go beyond robots.txt and implement stronger controls. Depending on your needs, these may include:

Network and server controls

• CDN and Edge Management: Use managed services like Cloudflare’s Block AI Bots to automatically identify and block traffic classified as AI training or agent behavior at the network edge.
• Server-Level Configuration: Enforce hard blocks using Apache (.htaccess) or Nginx configuration rules to reject requests from known scraping user agents, bypassing the voluntary nature of robots.txt.
• Web Application Firewalls (WAF): Deploy a WAF to implement IP address blocking for known scraper ranges, or set up rate limiting to throttle or drop connections from aggressive or unverified automated clients.

Application and verification controls

• HTML Meta Tags: Insert <meta name="robots" content="noai, noimageai"> tags into your page headers to signal, rather than enforce, content non-usage preferences directly to parsers.
• Behavioral Verification and Traps: Use advanced human verification like Cloudflare Turnstile instead of frustrating CAPTCHAs, or deploy “AI labyrinths” that feed infinite nonsensical links to bad-actor bots ignoring standard directives.

Legal and policy controls

• Legal and Licensing Terms: Update your site’s Terms of Service and Content Use policies to legally prohibit unauthorized data harvesting and establish a foundation for potential future enforcement.

No method is foolproof, but combining several layers of protection can reduce unauthorized crawling.

Limit AI influence

Controlling access to training data addresses only one part of the problem. The harder challenge is managing how AI outputs are presented to developers and end users, and how much authority those outputs receive.

AI outputs require context and framing. Content creators therefore still play a crucial role in explaining tradeoffs, showing evidence, and helping readers understand when an answer is incomplete. The value of content may shift from providing one more answer to teaching people how to investigate and evaluate answers.

Rethink roles and responsibilities

Instead of asking whether AI replaces developers, we need to ask how it changes their roles, skills, responsibilities, and career trajectories.

Developers will increasingly use AI tools to generate code, automate repetitive tasks, and assist with debugging. This shift adds responsibilities for orchestration, review, and debugging of AI-generated code.

AI does not remove the need for fundamentals or for humans to remain involved in critical decisions. Developers still need to evaluate tradeoffs, consider ethical implications, and verify the quality and accuracy of AI-generated output.

So, is it really an asteroid?

AI is a serious disruption to frontend development, but its impact is not predetermined. Whether it destroys or strengthens the profession will depend on how developers use and evaluate these tools, how publishers sustain trustworthy knowledge, and how platforms respond to the costs and responsibilities of AI-driven traffic.