32 points ankitvgupta 1 hour ago 7 comments
datadrivenangel 41 minutes ago | parent
Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
risyachka 25 minutes ago | parent
Fpga is software-driven, model can verify its output easily.
Zero surprise it did it
codekansas 19 minutes ago | parent
This is cool
Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...
chalkycrimp 6 minutes ago | parent
Good chance I'm not the only one thinking this: share the code