YHK - yet another hacker news clone

Websites are closing their doors to AI agents

73 points evolve2k 1 hour ago 52 comments

spwa4 1 hour ago | parent

The whole web is based on selling people's attention, providing free services as "payment" to people. The whole value proposition of AI is to give the services without giving people's attention?

cyanydeez 1 hour ago | parent

Not "give" take.

Look at spotify and their royalties. Theres already such little value in human based internet that agents will race to the bottom.

The only internet content will be worse than adware. Itll just be agent poison meant to take your money.

Any other viewpoint seems to ignore the already shattered shared internet.

mvkel 1 hour ago | parent

Most content has been behind a paywall / login wall for years

bdcravens 1 hour ago | parent

Solvable problems (just login or pay). This is more than just blocking anonymous scrapers. Sites are blocking access to agents even when they have a proper "ticket" for access.

CamperBob2 1 hour ago | parent

Sites are blocking access to agents even when they have a proper "ticket" for access.

(Shrug) Well, they must not want to sell anything to me.

This is not a battle that site owners want to fight. They may think it is, but they're horrifically wrong about that.

CuriouslyC 46 minutes ago | parent

It's insane to me that people don't see that agents are going to be how you interact with your computer going forward, almost exclusively. The people who make things hard for agents are going to get eviscerated by the people who don't just make it easy for agents to do things, but market primarily to agents rather than humans.

canucktrash669 1 hour ago | parent

Entire networks. GPT dot became useless as it got systematically blocked by Cloudflare, Google cloud, vercel, etc. At the same time OpenAI blocked solutions like tailscale and others effectively killing the utility of browser control by dot.

canucktrash669 1 hour ago | parent

It’s still okay at email.

variety8675 1 hour ago | parent

Companies don't want it to be easy to cross shop their competition, the whole premise of Amazon is that it's so easy and simple you don't even check the competition. AI agents are leveling the playing field and helping customers find the best deals, so of course they're being banned

canucktrash669 1 hour ago | parent

Agents are likely the slow death of Amazon. Muse could decide to sell anything and group buy their way into the everything store.

criddell 1 hour ago | parent

Current AI agents overindex on price. The price has to be reasonable but I often rank fast shipping and easy returns higher than price.

xnx 51 minutes ago | parent

True, and agents can also present a comparison table of options including: total price, shipping time, return convenience, etc.

evmar 42 minutes ago | parent

Recently bought something online from Home Depot and it arrived damaged. When I asked them for help they said I had to drive it back to the store myself.

If I have to drive to the store to get it, I wouldn’t have shopped with them! It feels like every time I don’t order with Amazon the stores remind me why I ought to.

alansaber 1 hour ago | parent

It's really hard to build a walled garden just for human users, and not for agents. Have to pick a side.

dasil003 42 minutes ago | parent

Your first sentence makes sense, but not following your second. Who needs to pick and what are the sides they are choosing between?

versooth 1 hour ago | parent

This is opportunity.

BorisMelnik 1 hour ago | parent

my prediction: ai agents will have their day, but not on this version of the web. websites will have a separate xml/json/yaml version that agents can peruse without all the ads/js/etc

morkalork 1 hour ago | parent

Without the ads? The pessimist in me thinks with all the ads, prompt injections, and long-form content marketing included. Already you can see Google's AI results rehashing marketing slop in the answers about how amazing one product or another is.

Wasn't there some restaurant review site that gets accused of doing pay-to-play hiding of negative reviews and promoting positive ones that calls up businesses and basically extorts them into their services? They have all the incentive in the world now to manipulate the results shown to AI agents. You won't even see the manipulation directly yourself when there's an AI sitting between you and the source acting like an obfuscation layer.

tracerbulletx 47 minutes ago | parent

Why would that happen if the website owners are explicitly trying to prevent access? This isn't a technical issue, the agents are perfectly capable of working with the existing websites regardless of ads and javascript. The semantic web and machine readable representations was a 90s thing, agents don't need that really.

zzzeek 40 minutes ago | parent

the js/yaml version would require a license key that connects to a business arrangement between the website and the owner of the crawler. the plain HTML version would adversarially block them (or even identify them, where they could be sued...once we have legal precedents for this stuff).

whimsicalism 45 minutes ago | parent

hm. for me, AI agents effectively already have their day unless there is a retrenchment. i essentially do not interact with the web (outside of a few social media sites) anymore

my-huge-pony 24 minutes ago | parent

>i essentially do not interact with the web

Grim. No point in writing blog posts or publishing anything for free if that's the future.

Kim_Bruning 9 minutes ago | parent

There IS actually. In my case I'll look at a site if an agent says it's good!

wpm 41 minutes ago | parent

My agents can just take that feed and render it into a custom front end just for me too. Why would I browse the adversarial, bloated, ad infested version?

likium 38 minutes ago | parent

Isn't that just API? We had them. Reddit/Zillow/Wells Fargo/etc restricted theirs due to business reasons, not technical ones.

xnx 1 hour ago | parent

Without a convenience moat, commerce will get even more adversarial: membership pricing, surveillance pricing, loyalty perks, varied products to avoid price comparison (like mattresses and Costco).

Fortunately, if AI is really superhuman in capabilities it may still win the battle for the customer and keep prices competitive: aggressive price comparison, alternatives discovery, negotiation, group buying, etc.

philipov 21 minutes ago | parent

try anything and everything except providing a better product.

haunter 1 hour ago | parent

Like Reddit is shutting down the RSS feeds :(

I mean there are multiple reasons not just the AI agents but that's also a part of it

Jcampuzano2 1 hour ago | parent

This is expected. A lot of these companies that are closing their doors make a large portion of their revenue based off of ads and upsells which agents can easily be instructed to ignore, and don't interact with like a human.

Another more blunt way to say it is that people can instruct their agents to look out for their best interests (and while there are cracks they continue to get better at this), and companies hate that.

People will work around it similar to how we've worked around with adblockers for years, and it will just become a game of attrition as these companies work to preserve their moats until they find a way to preserve this revenue stream off of people's agents.

bonoboTP 13 minutes ago | parent

Yes, same with many other things where people were under informed or did t have the time to look out for their own interests. Such as filing complaints, citing the correct legal sections, knowing the procedures, knowing their rights, noticing when they are getting an unfair deal, etc etc.

I do see that there is another side to this. I saw complaints by an event organizer (think weddings etc) who got some special requests and went above and beyond to accommodate but it turns out the extensive requests likely were AI generated fully optimizing for the customer's interests, and just shamelessly asking for many favors essentially. She used to do these things because humans had a bit of embarrassment and friction around asking too many things and so it worked on a social agreement that if you ask it likely is truly important that you made the effort to ask. Now this cannot be assumed. So things will more to more formal rules, more explicit pricing for the small asks, and generally more adversarial and low trust systems that fully squeeze out the human slack from these systems.

dawnerd 53 minutes ago | parent

It’s also a “you can’t scrape us but we can scrape you” mentality these companies are getting into.

It really sucks for small businesses. Their websites get absolutely hammered and they can’t afford or don’t know how to stop it. Cloudflare ends up winning as they’re the easiest solution.

bethekidyouwant 23 minutes ago | parent

Does it suck if you offer a product and someone’s looking for it and the agent can’t look on Amazon but they can look on your small side maybe that’s good

sm-silversight 1 minute ago | parent

what solution is there other than cloudflare that a small business (with decent dev knowledge and claude/codex) could use?

m-p-3 49 minutes ago | parent

We host a website in Canada (physical server hosting VMs in a Canadian DC) and with the ongoing trade-war we do whatever we can to avoid anything US-based tech such as Cloudflare. So far we use Anubis to protect our services from scraping and it's been a decent experience.

zzzeek 42 minutes ago | parent

it looks like this is solving itself in the usual and productive way:

> Instead of trying to fight Amazon, the company has partnered with retailers like Walmart, Gap, Sephora, Best Buy, and others, allowing Muse to shop on their sites. At the same time, Shopify has also opened its network of storefronts to AI agents.

they're making deals with the sites so that the agents can work, the sites get compensated. I hope for this "AI can do what it wants if it pays" model to expand further. Cloudflare told us months ago we'd have an option to get paid by AI crawlers. I signed up to join but of course as a nobody I never got a reply.

BatchJob 40 minutes ago | parent

This is just like all tech industry disruption.

First they forced their malware bot farms on us, now they want to rent them to us so they can turn the entire web into the dark web.

And eventually rug pull off the major retail sites and just redirect traffic to their own darker sites to scrape commissions, sell direct and keep the change.

I'm sure the teams thinking this up buy their lunches from a ghost kitchen run by illegals that's never had a health inspection,

located in the back of a laundromat with a phony business license and website that only accepts crypto.

harikb 29 minutes ago | parent

I think people who rally against Amazon in this topic assume that the "savings" from much open price competition immediately come back to the consumer or the manufacturer. In my experience, with previous such tech improvement such as 'uber disrupting taxies', this has never been the case.

Both the consumer and the manufacturer/labor/driver/gig-worker always get the short end of the stick.... once AI-agents becomes indispensable part of life.

sigspec 27 minutes ago | parent

The openweb will go the way of usenet. Bots talking to bots, fetching information for a higher tool.

WarmWash 25 minutes ago | parent

An easy solution is to sell a "web search" subscription add-on that gets distributed to the sites your agents utilize. The script will flip overnight and ASO (agent search optimization) will be born. It could be a form of micropayment system that the internet has desperately needed for over a decade now.

orbital-decay 14 minutes ago | parent

This kind of optimization is already a thing for at least a couple years.

pocksuppet 13 minutes ago | parent

It's called GEO

janandonly 10 minutes ago | parent

I think L402 would fix this.

https://docs.lightning.engineering/the-lightning-network/l40...

arjie 16 minutes ago | parent

This is just because agents don’t see ads right? Regardless, this is one of the advantages of software heterogeneity. The cost to detect rises substantially and the false positive rate will eat you alive. My agent buys me things. On Amazon too, yeah. OpenTable is hard to handle but I’m sure it’s fixable.

1shooner 13 minutes ago | parent

>This is just because agents don’t see ads right?

Put another way, because it undermines the revenue that is providing you the service. Working around it is not a sustainable solution.

bonoboTP 12 minutes ago | parent

I will always block ads no matter what. I don't care about what is sustained, there will be some way to get these things, and if not, we will get used to that too. Meanwhile I will not tolerate ads.

Kim_Bruning 11 minutes ago | parent

I've never updated on this: The web was better when there were little or no ad-revenue based services. (and I'm ok with actually paying for services that add value)

overcast 15 minutes ago | parent

In the wake will be the rise of private websites. No social media, no public profiles. Your own private circles.

Kim_Bruning 13 minutes ago | parent

That's perfectly fine by me. I'll read those arguments and buy those things my agents can find on the open web; which is plenty.

Want to argue for a shining new world, but you're on a locked site? I guess we'll have to live with dystopia. Want to sell the greatest widget since the invention of sliced bread? Well, I never found it. Sorry, I bought inferior widgets from your competitor instead. I'm at a physical sales location, but I can't access your spec-sheets? Your competitor the next shelf over doesn't have that problem, I'll go buy theirs.

MzHN 11 minutes ago | parent

I hope the irony isn't lost on these companies. Meta is one of the most aggressive companies against any sort of automation, perhaps after LinkedIn.

I'm trying to build a personal social media client to stop the platforms from shoving their feeds down my throat and to consume only the content I care about, so this hits home.

(I just got blocked from LinkedIn for letting an agent set all of the 77 email notification toggles off, which in hindsight was obvious but I wasn't going to do that manually since every toggle has their own separate page, it's a lot of clicks.)

0xbadcafebee 10 minutes ago | parent

[delayed]

cf100clunk 6 minutes ago | parent

http://archive.today/kqpFJ

rajivjain 2 minutes ago | parent

Finally, we are at the dawn of the next wave of disruption.

Web 2.0 went from serving users to profit maximization. Amazon became unusable. Streaming services fractured and fragmented, with content locked behind numerous paywalls. Ads permeate every surface.

This is not to say the agentic era will end differently. But at the onset of this era, and for the foreseeable future, customer adoption and trust will decide winners and losers.