4 users, batchedlong-context
32kcontext
741tok/s total
216.0tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.7.0 GiB resident687 MiB KV per user
did:epn:002408…e189d2 · q4_0 KVsigned
4 users, batchedlong-context
32kcontext
269tok/s total
78.0tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.7.0 GiB resident687 MiB KV per user
did:epn:002408…dae679 · q4_0 KVsigned
4 users, batchedhigh-throughput
8kcontext
178tok/s total
48.4tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.5.0 GiB resident171 MiB KV per user
did:epn:002408…c76d30 · q4_0 KVsigned
4 users, batchedlong-context
32kcontext
178tok/s total
48.4tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.7.0 GiB resident687 MiB KV per user
did:epn:002408…c76d30 · q4_0 KVsigned
4 users, batchedlong-context
32kcontext
173tok/s total
46.9tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.7.0 GiB resident687 MiB KV per user
did:epn:002408…a5be46 · q4_0 KVsigned
4 users, batchedlong-context
32kcontext
172tok/s total
46.2tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.7.0 GiB resident687 MiB KV per user
did:epn:002408…e61d3b · q4_0 KVsigned
4 users, batchedhigh-throughput
8kcontext
169tok/s total
46.1tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.5.0 GiB resident171 MiB KV per user
did:epn:002408…a5be46 · q4_0 KVsigned
4 users, batchedhigh-throughput
8kcontext
164tok/s total
45.1tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.5.0 GiB resident171 MiB KV per user
did:epn:002408…e61d3b · q4_0 KVsigned
4 users, batchedhigh-throughput
8kcontext
84tok/s total
56.3tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.5.0 GiB resident171 MiB KV per user
did:epn:002408…84bf54 · q4_0 KVsigned
4 users, batchedhigh-throughput
8kcontext
78tok/s total
49.6tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.5.0 GiB resident171 MiB KV per user
did:epn:002408…e189d2 · q4_0 KVsigned
4 users, batchedhigh-throughput
8kcontext
51tok/s total
29.3tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.5.0 GiB resident171 MiB KV per user
did:epn:002408…dae679 · q4_0 KVsigned
Single streamlong-context
32kcontext
45tok/s total
311.2tok/s · per user
One reply, streaming — the latency a single user sees.5.0 GiB resident687 MiB KV per user
did:epn:002408…e189d2 · q4_0 KVsigned
Single streaminteractive
8kcontext
45tok/s total
309.1tok/s · per user
One reply, streaming — the latency a single user sees.4.5 GiB resident171 MiB KV per user
did:epn:002408…e189d2 · q4_0 KVsigned
4 users, batchedhigh-throughput
8kcontext
30tok/s total
8.7tok/s · per user
4 replies decoding at once — the capacity this gram can actually serve a room.5.0 GiB resident171 MiB KV per user
did:epn:002408…9e3c79 · q4_0 KVsigned
Single streaminteractive
8kcontext
21tok/s total
109.7tok/s · per user
One reply, streaming — the latency a single user sees.4.5 GiB resident171 MiB KV per user
did:epn:002408…dae679 · q4_0 KVsigned
Single streamlong-context
32kcontext
21tok/s total
109.6tok/s · per user
One reply, streaming — the latency a single user sees.5.0 GiB resident687 MiB KV per user
did:epn:002408…dae679 · q4_0 KVsigned
Single streaminteractive
8kcontext
11tok/s total
75.6tok/s · per user
One reply, streaming — the latency a single user sees.4.5 GiB resident171 MiB KV per user
did:epn:002408…a5be46 · q4_0 KVsigned
Single streamlong-context
32kcontext
11tok/s total
78.7tok/s · per user
One reply, streaming — the latency a single user sees.5.0 GiB resident687 MiB KV per user
did:epn:002408…c76d30 · q4_0 KVsigned
Single streaminteractive
8kcontext
10tok/s total
134.5tok/s · per user
One reply, streaming — the latency a single user sees.4.5 GiB resident171 MiB KV per user
did:epn:002408…84bf54 · q4_0 KVsigned
Single streamlong-context
32kcontext
10tok/s total
78.6tok/s · per user
One reply, streaming — the latency a single user sees.5.0 GiB resident687 MiB KV per user
did:epn:002408…a5be46 · q4_0 KVsigned
Single streaminteractive
8kcontext
10tok/s total
77.4tok/s · per user
One reply, streaming — the latency a single user sees.4.5 GiB resident171 MiB KV per user
did:epn:002408…c76d30 · q4_0 KVsigned
Single streaminteractive
8kcontext
10tok/s total
76.6tok/s · per user
One reply, streaming — the latency a single user sees.4.5 GiB resident171 MiB KV per user
did:epn:002408…e61d3b · q4_0 KVsigned
Single streamlong-context
32kcontext
10tok/s total
76.1tok/s · per user
One reply, streaming — the latency a single user sees.5.0 GiB resident687 MiB KV per user
did:epn:002408…e61d3b · q4_0 KVsigned
Single streaminteractive
8kcontext
3tok/s total
11.7tok/s · per user
One reply, streaming — the latency a single user sees.4.5 GiB resident171 MiB KV per user
did:epn:002408…9e3c79 · q4_0 KVsigned