Yet another Hacker News

Amazon saves millions by replacing Sonnet 4.6 with Strands Decider

398 points aloukissas 6 hours ago 140 comments

oofbey 6 hours ago | parent

If their model costs 98% less than Sonnet, and Amazon is saving “millions” then these folks are raking in 2% of millions. Congrats I guess?

vincnetas 6 hours ago | parent

2% of millions ... so far. Take in to account growth potential.

Den_VR 6 hours ago | parent

small, open source, decision model optimized for fast experimentation, local development, and innovation...

But just think of the street cred!

mcintyre1994 5 hours ago | parent

I think you’re missing that Strands Agents are open source models released by AWS, not an external startup that Amazon is paying.

swiftcoder 5 hours ago | parent

Yes, this is more of less an advert, using their internal adoption as a case study

sant-sh 2 hours ago | parent

that's what it looks like. they are trying to showcase the usecase here

rrr_oh_man 5 hours ago | parent

I yearn for the day of going back to make v. buy, not buy v. buy

otterley 4 hours ago | parent

You're (understandably) a bit mixed up here. Strands Agents is an agent SDK. Strands Decider is a decision model.

sznio 6 hours ago | parent

why didn't they upgrade to further sonnet versions?

mikae1 6 hours ago | parent

How about at least reading a couple of paragraphs from the post this thread is about before commenting? :)

Zanfa 5 hours ago | parent

If they’re running on AWS Bedrock, lack of batch API support for more recent Anthropic models would be my guess.

coolgoose 6 hours ago | parent

Yea right... Paying market price of 3 usd per mil tokens on sonnet 4.6...

armcat 6 hours ago | parent

About six years ago we built a ticket classification/routing model and pipeline [1]. The initial version was trained using SVMs, it took a couple of minutes to train, and the ticket classification was just a few milliseconds. It's so fascinating to watch how the field moved over to LLMs, and then back to faster/cheaper classifiers again.

[1] "Linnaeus: A highly reusable and adaptable ML based log classification pipeline", https://arxiv.org/abs/2103.06927

NOTE: the work was actually accepted at the International Conference on Software Engineering (ICSE) in 2021.

floro 5 hours ago | parent

This is just the AI grift and uneducated hypebros. There is zero reason to use an LLM as a classifier. But AIbros with zero technical knowledge will push for it anyways even if it is 1000x slower and 1000x more expensive to run.

wodenokoto 5 hours ago | parent

You can can't build an SVM-classifier for a ticketing system without domain data. You can ask an LLM to do it, and it will do it very well.

With the LLM you don't even need a data scientist to train the model, you just need someone who can connect to APIs. And the LLM is fine with adding and removing categories. No retraining.

It makes the initial faces incredibly fast. Then it is slow and expensive to run, so once the system settles, it becomes the target for model optimization, e.g, a dedicated, or semi-dedicated model.

Karrot_Kream 4 hours ago | parent

I mean just have a coding agent do the whole process from start to finish. Tell it you want to seed an SVM from LLM runs on ticket data. I can bet that it'll do it all for you without much input from you at all.

gonzalohm 1 hour ago | parent

I think AI is making people forget that code can be used for automation.

I see coworkers using AI for tasks like "visit an API endpoint, download a file, put it in S3". This is so stupid. Okay if you want to use AI, but use it to write python code to do that, why pay in tokens every time you want to run that task, it makes no sense

IanCal 5 hours ago | parent

There are absolutely good reasons to use smarter things for classification. Svm with tdf idf for example is super fast but will make very bad classifications at times in ways entirely unfixable with that system (thinking back to bats being an acronym for a theology degree classification or similar)

NitpickLawyer 5 hours ago | parent

This is a bit too radical and misses some nuance. In order to start building classifiers you at least need some ML domain knowledge, like what to google at least. Meanwhile, point anyone at the jev/open-jev-clones API and have them start classifying in <10 minutes. There's some value there. (same used to be valid for LLMs before jev became a thing. You can use llms as zero-shot classifiers with the right API wrapper, w/ no domain data).

msp26 29 minutes ago | parent

Even if you have ml knowledge, it's still nice to be able to experiment with various use cases with zero upfront data/training time.

vidarh 4 hours ago | parent

I can spin up a classification solution with an LLM in ca. 5 minutes that is likely to do an okay job. To build a custom classifier, I need lots of training data first.

That is the reason.

Once you then have a system that is used enough, it starts making sense to look to replace that with a leaner classifier, but LLMs have a place in vast number of places where we'd previously never use a classifier because it was too much effort.

darkwater 4 hours ago | parent

> but LLMs have a place in vast number of places where we'd previously never use a classifier because it was too much effort.

Absolutely! But that doesn't rule out that there are AIbros that will tell you to use an LLM, even if the task could be done with a static analyzer with a few rules. Use an LLM because it's AI!

bgroins 35 minutes ago | parent

You seem to be quite fixated on a personality type rather than the underlying technology.

cryptonym 3 hours ago | parent

1000x more expensive run can be overall cheaper. You avoid upfront training/tuning cost. It all depends on your scale.

benrutter 5 hours ago | parent

Yeah, its pretty wild! I think one of the benefits, but also potential risks, with LLMs, is you can just point them at anything without a lot of expertise.

Building an SVM isn't a tricky job, but you probably need a data scientist involved, and to think carefully about a labeled dataset.

I think we're seeing the modelling equivalent of something like electron apps. Where they waste a tonne of resources, but make an easier developement experience.

Will be interesting to see how much or little this changes with LLM costs.

IshKebab 4 hours ago | parent

> Building an SVM isn't a tricky job, but you probably need a data scientist involved, and to think carefully about a labeled dataset.

Well not anymore... You can just get AI to build it.

benrutter 3 hours ago | parent

True I guess. I think most companies would at least want sign off from someone who's heard of SVMs that it isn't doing something mad.

Just like, LLMs can build web back ends, but I'd want a security professional to give it some kind of thumbs up. Especially if a lot of money rides on it.

Foobar8568 3 hours ago | parent

I have seen Phd in stats validating models with data the models have been trained on. And I don’t live in a country with dubious universities.

So between llm and pseudo experts, at this stage, I prefer to go with a LLM.

jval43 5 hours ago | parent

SVMs in big data were already well liked back in 2010 as they could be scaled via map-reduce.

An issue in industry was that companies would not allow training on their data across customers. Understandably so, as SVM dimensions are readily interpretable given some thought. General purpose datasets were ill-suited too depending on the task.

So you'd start from scratch every time, annotate features like crazy, and often didn't have enough training data overall.

LLMs "solved" that by just ignoring the previous rules and training on everything, which lets the necessary world knowledge seep back in.

I'm sure we'll see many combinations of LLMs and regular machine intelligence in the near future.

exe34 4 hours ago | parent

And you can now get that world knowledge out of llms and generate synthetic data to train your svm and fine tune on your own dataset!

flatline 3 hours ago | parent

I’ve hand-labeled so many freaking datasets over the years for supervised learning tasks, if only to build out a PoC. Would 100% hand this off to an LLM and just fine tune the results. I definitely do not want an LLM in the loop at inference time for anything I could build a simple classifier for.

IanCal 2 hours ago | parent

Llm labelling is so incredibly useful, as is transforming labelled data to other valid but different forms (like where you might train on sentences but also randomly replace days of the week with others).

You can also do stuff like setup active learning but have the labeller be an llm as well to make things more efficient.

Definitely worth it for PoC level things if not full ones.

Edit - also reviewing, so it can run training, test, review failure cases and investigate itself. You can leave them to plough through a few iterations where the early part is the classic time you’d have a few obviously bad sets of results.

miki123211 2 hours ago | parent

LLMs, and other transformer models before them like BERT for example, have the benefit of "transfer learning."

If you want to make a ticket classification model, you don't need to train across many customers doing ticket classification, who might not want you to train on their data and whose data is impossible to obtain unless you already are a large company. Instead, you can train on a completely different task which forces the model to learn the syntactic and semantic structure of the English language — say predicting missing words in Wikipedia articles, which you can't do successfully without a good understanding of English — and then fine-tune that model on your specific task with much less data.

Because the initial task is so general, you can have one company do the expensive pretraining, share the base model, and then have everybody else just do the finetuning on their (small) proprietary dataset.

LLMs and models like Jev go one step further, letting you use them as classifiers with no training at all, with nothing but rules in the prompt. And with a few tricks, you can indeed use them to get pretty high-quality labels, which you can then distill into a fine-tuning dataset for a much smaller model.

thesz 2 hours ago | parent

  > say predicting missing words in Wikipedia articles, which you can't do successfully without a good understanding of English

This is word2vec model, one of - predict a word from context. The other one is to predict surrounding words (context) from a word. It did a good job when I used it, words' classes were impeccable for most practical purposes.

Word2vec is a two layer perceptron.

mrbungie 1 hour ago | parent

¿Por qué no los dos? Big skill issue, you can even route known problems to a trained in-house simple classifier and low-confidence problems to Jev-like "zero shot classifiers" or LLMs.

I feel like people in the field forgot everything.

cogogo 22 minutes ago | parent

It will only get worse. Non-tech people will be (or are) very suspicious of anything trained in house. The bar to spend time/money on anything domain specific will only get higher.

abhgh 2 hours ago | parent

Agree to all of this but I would make a minor correction: the interpretability is true under certain conditions, e.g., you use something like n-grams and train an SVM with a linear kernel. You could change the representation back then too, e.g., LDA (topic models), Latent Semantic Indexing, and of course, you could change the SVMs kernel as well, e.g., use a sequence kernel [1]. Some of these were much less interpretable. But the common pipeline back then was indeed n-grams + Linear SVM. Today it is easy to modernize the pipeline by just changing the representation step to use an embedder like ModernBERT. I have a "one page" classifier if you want to try it out - [2].

[1] I went down the rabbit hole of using sequence kernels quite a bit back then, see https://arxiv.org/abs/1910.13800, but the timing was terrible. RNNs and variants like LSTMs were beginning to become popular, and everyone (including me) switched.

[2] https://pocket-text-lab.abhishek-ghose-82.workers.dev/

chaostheory 5 hours ago | parent

The tech industry is not that far off from the fashion industry

autuni 5 hours ago | parent

I think this kind of change will pop up more often.

In software engineering, we learned a long time ago that giant monoliths are not the answer. You want smaller, but specialized units that work together. They perform better at their specific task, are more maintainable, you can just upgrade one part of the system without having to impact the others, ....

I don't know where the focus on building a single model that can do everything comes from.

Geezus_42 5 hours ago | parent

I was thinking about this recently as well. Smaller, specialized models that can run on device just make so much more sense to me than these huge generalized models that need huge infrastructure run.

ivell 4 hours ago | parent

I am not sure a best practice in one area translates nicely to another. LLM's power comes with it being a monolith.

autuni 4 hours ago | parent

Does it though? Let's say you use an LLM to write code, do you think it benefits from the fact that the same model is capable of translating corporate messaging into internet slang for marketing?

I would assume a smaller model trained specifically to produce code based on natural language input would perform at least just as well, which would already be better given the smaller size / probably lower training resource requirements. This is of course, as already said, an assumption. But generalizing typically comes at a cost, there is no reason why this wouldn't apply to LLMs, it even applies to humans.

heggy 3 hours ago | parent

Isn't this just the Bitter Lesson over and over again? Specialised approaches get outperformed by more compute and a simpler general approach pretty much every time.

try-working 2 hours ago | parent

Bitter lesson is wrong. Large general models are outperformed by small specialized models.

9rx 1 hour ago | parent

The Bitter Lesson makes no distinction between large general models and small specialized models, assuming they are both trained from data.

The Bitter Lesson is that human-written rule models will always lose out to trained models in the end.

IanCal 2 hours ago | parent

Or would it be worse as it wouldn’t have the broader knowledge to understand the reason you’re coding what you are anyway, understand the logs and iterate?

Coding an exercise app will benefit from a model that can code and understands what exercise is. A car showroom website will benefit from a model that understands what relevant filters would be.

autuni 1 hour ago | parent

Of course there are synergies between different fields. I'm not saying cutting everything off as close as possible to it's purpose must be the best way. But that's a long way from a single generalized model.

ianjbutler 53 minutes ago | parent

> not saying cutting everything off as close as possible to it's purpose must be the best way

Nah, let’s stand firm here and not give in so easily. Parent suggests a coding model needs exercise info in case you want to code an exercise model? Absurd, that leads to the know everything monolith to do anything.

Language, reasoning, tool use, and domain knowledge.. in this case the domain is coding. Someone wants an exercise app? Great, so we look that shit up, we don’t bake it in. We don’t want to bake anything we don’t need in.

We don’t even need to explain this completely reasonable and totally normal engineering preference that small is beautiful. it’s the other side that needs to justify wtf exercise has to do with coding or why everything from twitter posts to a working knowledge of 15th century French poetry just might be helpful for disconnected domains, like say serious work in contemporary astrophysics. Back to basics, No one asked for that, No one should accept as given that this is the way anything needs to work

IanCal 14 minutes ago | parent

> Parent suggests a coding model needs exercise info in case you want to code an exercise model?

No, the point is having broader knowledge than just "can code" is important for building software for humans to use.

> We don’t even need to explain this completely reasonable and totally normal engineering preference that small is beautiful.

It is, of course it is, as long as that small thing is useful. My argument is not "let's put literally everything in this thing just in case" it's "broader world knowledge is important for building many things".

Also, we don't look at CVs or histories of people and say "hmm, this person could be good at coding for us but I really wish they didn't have such a broad and expansive history".

> Great, so we look that shit up

But what do you have it look up? What is exercise? What is a muscle and what is a human? How is it going to understand those references? Does it have the broad enough understanding to see where the gaps are?

At some level unless all you want is "loop over the array and sum the results into a variable called sales_total" converted into code, you have to have some broader knowledge about things.

And once you start considering that maybe it needs more than simply code, we're not disagreeing we're just discussing where the line should be. Which I've also not placed, because there's no simple statement to make on that.

duskdozer 1 hour ago | parent

They need some of that fluff to keep up the illusion of intelligence and sycophancy that gets people to rely on them. Additionally, they need to make sure they don't prune so much that the model size seems manageable on consumer or standard business hardware, otherwise more people may become tempted to self-host a distilled or other semi-open model. The model size is also a marketing point.

9rx 3 hours ago | parent

> LLM's power comes with it being a monolith.

Does it? All of the latest frontier models that have made technical details publicly available are using MoE; smaller specialized units that work together.

nixon_why69 2 hours ago | parent

Yeah but they're all trained together along with their gate tensor. Evaluating a moderate length prompt pulls in most/all of them.

darylteo 1 hour ago | parent

> single model that can do everything

"AGI"

pletnes 4 hours ago | parent

Interesting! I had the idea to compute embeddings of log lines to look for “black swans” in my logs. A small model running locally would do. Then PCA, SVM - any old school method should work, I guess. Haven’t seen it done but it must be tried by someone somewhere.

solidasparagus 4 hours ago | parent

What has changed is that expensive models can be deployed instantly and generate training data until you swap over to training your own cheap models.

Your model only took minutes to train if you don't count the data collection and labelling work

elvircrn 3 hours ago | parent

Svaka cast :)

matthorse 3 hours ago | parent

It's pretty interesting how a very broad field of machine learning got reduced to only LLMs for most consumers. Probably it's due to marketing (and of course how powerful these models are), but their lack of capability in many tasks already forces people, even amateurs, to use LLMs to write deterministic code instead. I wonder if the sudden appearance of viral classifiers predicts more common use of smaller ML models as the next step?

miki123211 2 hours ago | parent

non-LLM models just aren't visible to consumers.

Saying that the field got reduced to only LLMs is like saying that the field of combustion engines got reduced to cars. The trucks, combine harvesters and peaker plants are still here, they're just in the background, invisible to people who are not part of the industry.

Banks still use random forests and SVMs, medicine and image processing still use CNNs (and people who take photos with their phone don't even realize how much image processing is being done), TTS, ASR and predictive typing models are in wide use, and there's probably still plenty of BERT in spam classification and such.

armcat 1 hour ago | parent

I've seen a few people comment along the lines of "yes but LLMs dont need lot of data nor competence". So with our classifier "Linnaeus", the data came from JIRA tickets plus additional signals. To get up and running we didn't use a lot of data (a few dozen tickets across approximately 20 classes to begin with - eventually expanded to 120+ classes). There was very little to no manual data labelling - mostly it was devs selecting "gold standard" JIRA tickets for us to ingest. The model was retrained daily, and we had extra signals like thumbs up/down with comments that would be used for re-training.

tjungblut 17 minutes ago | parent

I have replaced a few of my LLM pipelines with gemma4 embedding and a classifier/regression head now. LLMs are actually really good to gather the data and label it with only prompting, but after a while it is just more economic to train your own model.

phillennium 6 hours ago | parent

It's funny how AI sometimes feels like it went full circle:

— "Let's build efficient classifiers"

— "Forget about the silly classifiers, LET'S BUILD A MACHINE GOD"

— "Ah, that machine god is often too expensive and underwhelming. Let's build efficient classifiers"

Schiendelman 5 hours ago | parent

The key is that the circle takes a few weeks instead of a decade.

written-beyond 4 hours ago | parent

You'll be surprised how fast things move when there's European GDP levels of investment going into an industry.

oblio 3 hours ago | parent

You should probably say "European country GDP levels of investment" as the European economy as whole is +/- the size of the US economy.

solidasparagus 4 hours ago | parent

If you were actually involved in that work pre-LLM, you know it didn't go in a circle. Most projects died at training data and problem definition. That isn't the case with LLM applications which can ship with zero data collection and labelling effort.

Now you can iterate on your 20 ticket categories in days against real workloads and not commit to training an efficient model until you've proven the prototype in production.

dejj 1 hour ago | parent

"Let them eat cake"-and-find-out-phase is what I'd like to call step 2.

Kaedon 6 hours ago | parent

At ~$21,700 saved per million tickets, saving millions annually would suggest at least 92 million tickets a year to reach $2 million saved. Is that about right?

I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.

f0cus10 5 hours ago | parent

yes, there's a ticket for EVERYTHING. people have tickets cut in their names even decades after they've left

fy20 5 hours ago | parent

At my company we also do the same every request to engineering (alarms, security, bug, customer support escalation) gets a ticket.

f0cus10 4 hours ago | parent

i mean there's even tickets cut if you need IT to fix a laptop or assign you a locker or unlock the code on your desk ped.

heck, there's even a ticket if the office toilet is clogged

tippa123 4 hours ago | parent

I once had to raise a ticket for a double AA battery for my mouse. This sort of stuff frustrated me enough to start my own company.

baxtr 4 hours ago | parent

Were there tickets for tickets too?

tippa123 1 hour ago | parent

Yes, plus let's close this ticket and open a new one so that the KPIs look solid.

phillc73 4 hours ago | parent

What does your company use for ticketing?

tippa123 1 hour ago | parent

Just spreadsheets for the time being. I think irrespective of the tool it's more the mindset for me. What is too small to have meticulously documented versus a general bucket. Also what becomes too big to not track on a ticket anymore but via project governance.

hyperbovine 1 hour ago | parent

Ah! You must be that energy harvesting mouse company.

kakacik 3 hours ago | parent

If it works it ain't stupid. Big companies desperately need traceability and ownership. Way too many bullshitters playing politics, being loud on meetings etc but very little actual output.

oblio 3 hours ago | parent

In some big companied many core corporate teams which hate that level of transparency and build their own opaque systems. I've seen this for departments like IT, HR, legal, etc.

I'd rather have tickets for everything and full transparency except for confidential info.

iamflimflam1 2 hours ago | parent

Im a huge believer in transparency. But I also understand why, in a big company, people don’t like it. It’s very easy for metrics to be weaponised by political players who are out for themselves.

vasco 5 hours ago | parent

Well if they use VPN point to point connections managed by AWS they'll get an email per week until they die for each VPN instance from AWS that they need to put in trash so I understand them.

sjdh76 5 hours ago | parent

Well there is the UFO/Bigfoot/Poltergeist crowd.

If you dont fully pay attention to their theories about what they think is the reason for the suspicious stain on the ground in the parking lot, the queue is going to get flooded. If your Boss belongs to that club oh man god say the division.

iLoveOncall 3 hours ago | parent

Yes, easily. I work for a "small" org in Amazon and it has more than 100,000 tickets a month. And it's a small fish in terms of number of tickets. As another commenter pointed out, tickets are used for basically everything administrative.

I checked just for fun and there are actually a LOT LOT more than 92M tickets in the past year.

catchnear4321 1 hour ago | parent

it is a corporate mantra many of us want to forget. it also functions as a response to any request.

“submit a ticket!”

a012 1 hour ago | parent

When you’re enterprise size, every request needs a ticket otherwise it didn’t happen

mathgeek 44 minutes ago | parent

Exactly. For folks working at smaller companies, think of it as "if you'd Slack someone to ask them to do something, that's a ticket" just purely because Slack doesn't scale to dozens of people needing to figure out which person even needs to get messaged in a given minute of the day.

tdhz77 5 hours ago | parent

You can do this even more efficiently with the rust crate bonsai. You will have more control and you can create sceneries of logic. You can use incredibly cheap models this way and get way better results.

I built a real time character story. In which players interact and integrate fictional characters. There is no delay in response and the characters are all in character. Oh and it’s all voice.

It’s shocking how effective this is.

soon_to_be_sb 5 hours ago | parent

Where can I read more about this!?

golf1052 4 hours ago | parent

I don't see how "bonsai + cheap model" is any better than the solution they used which was just "cheap open-source decision model that can be run internally at Amazon".

creativeSlumber 2 hours ago | parent

do you have a write up of this?

lencastre 5 hours ago | parent

counting the miliseconds until the insurance industry picks this up

3eb7988a1663 5 hours ago | parent

Classifying into one of 20 groups? Did anyone try scikit learn?

lofties 5 hours ago | parent

I was going to say, and I say this a little bit too often, a BERT model can do this even faster, with even less memory...

aitchnyu 1 hour ago | parent

Heard about classifiers a long time back. Will current ones resolve "reallyyyyyy beast mode" as a positive phrase like an LLM?

nlpnerd 5 hours ago | parent

Founder of Seldon AI (www.seldon-ai.com) here. We are working on dev tools to help developers with this kind of refactoring and continual production work.

Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.

Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.

The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.

They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".

These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.

I think system one models is a sensible paradigm, and we are on the precipice of something great, but there is a lot more work needed above the model layer to make this work.

Rapzid 26 minutes ago | parent

@dang why can't I down vote this?

Asking for a friend.

No, serious question; there are no downvote controls on this "advertisement", what gives?

hubertzhang 5 hours ago | parent

People use machine learning models before and after LLM is invented.

cmiles8 5 hours ago | parent

This pattern is playing out a lot. Running LLM inference at scale on apps typically lacks ROI. But I increasingly see folks use LLMs to help improve classical techniques.

So there’s some LLM use to develop a new version of algo, but maybe 2% of what would have been used by putting LLMs in the path of every API call. That generally does have ROI.

Putting LLMs in the loop each time is generally a dumb move, unless you’re an AI bro trying to get everyone to use LLMs for all the things.

psnehanshu 5 hours ago | parent

Ok, but why did they chose Sonnet in the first place? Couldn't they have chosen Haiku or GPT Luna or maybe even some small cheap Chinese model?

theshrike79 4 hours ago | parent

I’m guessing accuracy. Smaller models are just THAT much more inaccurate to not make it viable.

psnehanshu 2 hours ago | parent

I think for this purpose a small model will do reasonably well

jeremyjh 7 minutes ago | parent

Right. So we can either assume the engineers at Amazon are fucking clowns who don't do any engineering, or we can assume they tested and found that Sonnet was the cheapest model that met their reliability targets.

If you think the former is a likely hypothesis, then why did you even click on TFA?

psnehanshu 1 minute ago | parent

Let's keep the discussion civil

Hamuko 4 hours ago | parent

Not sure if it would even need to be a small Chinese model. Amazon would surely have the necessary hardware and infrastructure to run a bigger Chinese model with a low per-token cost too. They probably get their Sonnet at a discounted rate too since they do the inference "locally", but I imagine the savings for not having to pay Anthropic their share would still add up.

loafoe 3 hours ago | parent

Strands decider actually uses a small cheap/free Chinese model for its base architecture: Qwen

llukas 4 hours ago | parent

Are developments like that priced in Anthropic IPO?

nikit1411 4 hours ago | parent

yah i mean older models dosent mean they are worthless i still use cluade opus 4.6 for day to day task

speedboat199 4 hours ago | parent

Meanwhile they could have just switched to Haiku 5.5 to get same savings. No need for complex orchestration involving GPU instances and would have just been a one line change in the code.

Oras 4 hours ago | parent

Nice marketing to sell their decision model.

gvv 4 hours ago | parent

What I don't understand is why are they not offering the decision model via Bedrock yet. Sagemaker / Lambda are quite expensive if you're not using the model continuously.

Gasp0de 4 hours ago | parent

This article is so confusing. For example, they say they tried the Strands Decider model and dismissed it for Gemma, then they continue to call the model they used Strands Decider? Did I miss something?

ahoka 4 hours ago | parent

Not exactly sure, but as I understand this version of Strands is Gemma fine tuned for decision.

ninjalanternshk 2 hours ago | parent

This article explains it better (and rather colorfully)

https://strandsagents.com/blog/introducing-strands-decider/

They lopped the head off an existing Qwen or Gemma model and used its torso with their new head.

MisterMunchkin 1 hour ago | parent

It appears they just re-label other people’s models and call it a decision model.

egorfine 4 hours ago | parent

> JSON key order changed the answer. Sending the same content with keys in a different order moved the output probabilities by up to 0.19

Isn't it detrimental?

0x00cl 4 hours ago | parent

Sounds like a bunch of `if` statements with extra steps. Maybe throw an RNG decider too.

andai 4 hours ago | parent

50x cheaper, damn. I recall Chinese LLMs being similarly cheaper (and quite suitable for such simple tasks).

I think a decision model is superior here based on the task, and of course they weren't out yet when they started, but I'm wondering if they wouldn't have similar results (on price, probably not on speed) if they went with DeepSeek or something?

I don't mean send the data to china obviously — there must be an American provider that offers it?

Anyway it's moot in this case (decision model is a better fit) but it just makes me wonder who else is overpaying by 2 orders of magnitude.

a1o 2 hours ago | parent

It does use qwen, just check its repository linked above

tveita 4 hours ago | parent

> One thing that surprised us was that JSON key order changed the answer. Sending the same content with keys in a different order moved the output probabilities by up to 0.19.

That's like... really basic LLM knowledge? Like how e.g. this article measures how often answers change based on order https://zachlandes.com/trolley-problem/#appendix-a:~:text=Ch...

Or is this the Claudish "One thing that suprised us" that just means "and here's another thing"?

creativeSlumber 2 hours ago | parent

I did not know that json order can change the answer. Doesn't that make LLM's unreliable? Do we know why this happens?

xolox 1 hour ago | parent

I think this is similar to how small changes in phrasing can completely change the answer you get from an LLM. This all goes to confirm that LLMs are, at their core, still text completion engines, they (frequently / always?) don't have a firm semantic grip on the topic at hand.

Here's an interesting reference on this that gives a bit more background information:

-- 8< --

Because LLMs (and many other ML systems) are chaotic, it is possible to manipulate them into doing something unexpected through a small, apparently innocuous change to their input. These changes can be illegible to human observers, which makes them harder to detect and prevent.

For example, flipping a single pixel in an image can make computer vision systems misclassify images. You can replace words with synonyms to make LLMs give the wrong answer, or introduce misspellings or homoglyphs. You can provide strings that are tokenized differently, causing the LLM to do something malicious. You can publish poisoned web pages and wait for an LLM maker to use them for training. Or sneak invisible Unicode characters into open-source repositories or social media profiles.

-- >8 --

Source: https://aphyr.com/posts/412-the-future-of-everything-is-lies...

simianwords 3 hours ago | parent

why didn't they use Jev or Luna decision model? Seems odd that they used Sonnet and Strands instead of trying out of the box solutions

win311fwg 2 hours ago | parent

Because when Sonnet was originally chosen, Jev/Luna weren't available to choose from? The decision model movement hadn't happened yet. When they were ready to replace Sonnet, they already had their own decision model. Why use someone else's?

origini 3 hours ago | parent

So why did they not even try Haiku 5.5?

Why would Amazon go to another company to let them provide an adjusted decider model?

Amazon already has all these tickets labeld, they could just have build it themselves?

hobofan 3 hours ago | parent

What other company? Strands is Amazon (AWS).

origini 3 hours ago | parent

Oh i see.

Never heard of Strands before, felt complelty different when reading it more like a sales pitch from a consulting company

T4iga 2 hours ago | parent

They just didn't say. I think we can safely assume they are not completely stupid. They probably tested a few models a few months ago and have been using Sonnet since as it passed their benchmark/performed the best and then didn't change the system since, because it worked.

The blog post was focused on the switch to a decider model, not the initial architecture.

I think I'd be foolish to 'why didn't they just?', on the basis of <blank> because no mention was made in the post.

ZiiS 3 hours ago | parent

Amazon saves millions by asking their pencil manufacture to make them 1mm shorter. At that scale anything adds up to millions.

nalekberov 3 hours ago | parent

As a customer I don’t care what model they are using. Amazon is one of the worst e-commerce platforms (at least here in Germany) - damaged boxes, damaged items, items taking forever to ship, having to deal with rude delivery people just to name a few.

This is perfect example of how to not approach solving business problems. What do they actually need? Listening to customers.

What are they doing? Outsourcing delivery to third parties, lowering quality controls, creating worst working conditions, filling the platform with sponsored listings and ads, allowing any sort of cheap Chinese garbage with fake 5 stars on the platform, making the UI worse and worse everyday, and the list goes on and on…

iLoveOncall 2 hours ago | parent

> Amazon is one of the worst e-commerce platforms (at least here in Germany) - damaged boxes, damaged items, items taking forever to ship, having to deal with rude delivery people just to name a few.

I lived in Luxembourg for 4 years, which you have to order through amazon.de and is fulfilled by amazon.de, and I've never had any of the issues you describe. And that's with an average of 120 orders per year.

nalekberov 2 hours ago | parent

Just because you aren't running into problems doesn’t mean the platform is perfect; it might simply mean your standards are lower.

Use some logic before attacking: Luxembourg is only the size of Saarland (German state). Fewer orders and fewer deliveries naturally mean fewer problems.

iLoveOncall 2 hours ago | parent

> Fewer orders and fewer deliveries naturally mean fewer problems.

How about you read before attacking? I said it's pooled with all german orders.

OtomotO 2 hours ago | parent

If you use the right tool for the job at hand, you save money.

Of course you can try to dig a hole with a hammer instead of a shovel...

Rapzid 1 hour ago | parent

Slop article. First paragraph is 100% AI! Fucking lazy ass slop drivel.

docheinestages 1 hour ago | parent

No mention of quality, or at least I didn't see it. What seem to matter nowadays are speed and cost.

pluc 56 minutes ago | parent

For 1 measly million I will say yes or no to anything.

bilekas 44 minutes ago | parent

I'm not very up to date on the Ai lingo, but is a Strands Decoder just a decision tree in the same vein as Jev ?

mdnahas 41 minutes ago | parent

I am assuming that Strands Decoder is similar to Jev.

I do not believe Jev is a “decision tree”.

bilekas 9 minutes ago | parent

Should have rephrased it as a "decision model" it's all a bit confusing terminology to keep up these days. "Noul" I guess!

milimetric 11 minutes ago | parent

ooh, this is gonna hurt later:

"except that on long tickets Strands Decider drops the oldest comments to fit."

milimetric 11 minutes ago | parent

ooh, this'll hurt later:

"except that on long tickets Strands Decider drops the oldest comments to fit."