金のニワトリ (@gosrum)Mac Studio(M2 Ultra)에서 llama-bench로 Mistral-Medium-3.5-128B의 추론 속도를 측정한 결과를 공유했다. prefill은 약 57.5 tps, decode는 약 6.5 tps였고, 128k 컨텍스트는 가능했지만 200k에서는 OOM이 발생했다고 밝혔다. 대형 모델의 로컬 실행 성능 한계를 보여주는 테스트다.https://x.com/gosrum/status/2050706398070354085#mistral #llamabench #inference #llm #benchmark
Related
The departure of Jensen Huang from leadership of NVIDIA has huge implications for the AI industry. It appears that Jense...
The departure of Jensen Huang from leadership of NVIDIA has huge implications for the AI industry. It appears that Jensen was backstopping the circular financing etc.Ed Zitron poin...
Google、安くて速い「Gemini 4 Hydrogen」。AI指標はArgon上回るhttps://pc.watch.impress.co.jp/docs/news/2156118.html#impress #市場 #AI #Gemi...
Google、安くて速い「Gemini 4 Hydrogen」。AI指標はArgon上回るhttps://pc.watch.impress.co.jp/docs/news/2156118.html#impress #市場 #AI #Gemini
【笠原一輝のユビキタス情報局】フアン氏辞任のNVIDIA、33年の成功を支えたCUDAと次のリーダーの条件https://pc.watch.impress.co.jp/docs/column/ubiq/2156149.html#impres...
【笠原一輝のユビキタス情報局】フアン氏辞任のNVIDIA、33年の成功を支えたCUDAと次のリーダーの条件https://pc.watch.impress.co.jp/docs/column/ubiq/2156149.html#impress #半導体_周辺機器 #GPU #NVIDIA #市場 #動向 #その他 #AI