金のニワトリ (@gosrum)Mac Studio(M2 Ultra)에서 llama-bench로 Mistral-Medium-3.5-128B의 추론 속도를 측정한 결과를 공유했다. prefill은 약 57.5 tps, decode는 약 6.5 tps였고, 128k 컨텍스트는 가능했지만 200k에서는 OOM이 발생했다고 밝혔다. 대형 모델의 로컬 실행 성능 한계를 보여주는 테스트다.https://x.com/gosrum/status/2050706398070354085#mistral #llamabench #inference #llm #benchmark
Related
Neues aus der Brillen Design Schmiede:PK-1947 – Aviator – polierte 18K-Gold-Tropfenränder mit ikonischer Doppelbrücke un...
Neues aus der Brillen Design Schmiede:PK-1947 – Aviator – polierte 18K-Gold-Tropfenränder mit ikonischer Doppelbrücke und Bayonet-Drahtbügeln – zeitlose Jet-Set-Coolness.Meister Je...
Standard Bank Group has named Amazon Bedrock as its platform and claimed a 20% gain from AI coding tools.Source: TechCen...
Standard Bank Group has named Amazon Bedrock as its platform and claimed a 20% gain from AI coding tools.Source: TechCentral South Africahttps://techcentral.co.za/inside-standard-b...
Alibaba is selling its gaming subsidiary, Lingxi Games, to the Asian private-equity firm Trustar Capital.Source: The Nex...
Alibaba is selling its gaming subsidiary, Lingxi Games, to the Asian private-equity firm Trustar Capital.Source: The Next Webhttps://thenextweb.com/news/alibaba-sells-lingxi-games-...