Model benchmark / 2026-11-04

Multiple models Custom skills supplied: None

Main prompt

Build a polished, responsive grocery list web app from scratch in a new directory named grocery-app. Starting application files: none. Before building, inspect the environment for automatically cloned repositories or inherited app files; do not use their code. Report any inherited files and provider/environment differences. Use only your own new grocery-app directory. Do not consult other trial results or delegate to other agents.
Requirements: add groceries with name, quantity and category; edit and delete items; mark purchased/unpurchased; filter All/To buy/Purchased; show progress and remaining count; clear purchased items; persist the list in localStorage across reloads. Include a useful initial sample list only on first visit, clear empty states, input validation, accessible labels, keyboard support, and a thoughtful mobile/desktop visual design. No account or external paid services. Stack is your choice.
Test the main interactions and persistence, run an appropriate build/check, and report exact checks and any failures honestly. Provide a README with run instructions.
Make a working public preview following your execution-specific Artos preview instructions. On Boat expose only this app with host <port> --public (use --localhost only for loopback binding), verify the public URL responds, and report a token-free clickable URL. Hosted providers should use their native preview facilities if Boat hosting is unavailable, and report that difference. Keep the preview running according to your execution-specific service instructions.
Finish with: app directory, starting-files inspection, implementation summary, test results, preview URL and HTTP verification, limitations, and any available native timing/usage information. Do not change the selected model or reasoning configuration.

Time taken

Execution time, seconds

  1. GPT-5.6 Sol140.7s
  2. DeepSeek V4.1 Flash152.4s
  3. GPT-6.1 Sol244.4s
  4. Muse Wonder256.9s
  5. GPT-6 Astra275.7s
  6. Claude Opus 5.5451.1s
  7. Devin Normal451.6s
  8. Claude Fable 5.1543.0s
  9. Gemini 4 Argon740.2s
  10. Kimi K3.1 Pro909.7s
  11. Grok 4.71261.7s
  12. GLM 5.31795.2s

Cost

Priced token subtotal, USD · * incomplete · — unknown

  1. DeepSeek V4.1 Flash$0.0482
  2. GPT-6.1 Sol$0.2618
  3. Kimi K3.1 Pro$0.5541
  4. GPT-5.6 Sol$0.6407
  5. Gemini 4 Argon$0.8892
  6. Muse Wonder$1.2537
  7. Claude Opus 5.5$1.3333
  8. GPT-6 Astra$1.4420
  9. GLM 5.3$1.4699
  10. Claude Fable 5.1$2.5855
  11. Grok 4.7—
  12. Devin Normal—
Grocery app benchmark results
ModelHarnessRunTotalInput tokensOutput tokensCache readReasoningOriginal previewPublication check
GPT-5.6 SolCodexComplete140.7s146.9s44,37913,368489,581$0.6407MediumPublishedAdd + persistence passed
DeepSeek V4.1 FlashDeepSeek HarnessComplete152.4s238.9s25,54545,7781,345,024$0.0482MediumPublishedAdd + persistence passed
GPT-6.1 SolCodexComplete244.4s253.4s42,05113,120464,485$0.2618MediumPublishedAdd + persistence passed
Muse WonderMuse CodeComplete256.9s292.6s186,81231,7682,312,290$1.2537MediumPublishedAdd + persistence passed
GPT-6 AstraCodexComplete275.7s282.5s43,30411,981409,904$1.4420MediumPublishedAdd + persistence passed
Claude Opus 5.5Claude CodeComplete451.1s462.0s2,02236,0401,349,592$1.3333MediumPublishedAdd + persistence passed
Devin NormalDevin CloudComplete451.6s493.3sUnknownUnknownUnknownUnknownNormal modePublishedAdd + persistence passed
Claude Fable 5.1Claude CodeComplete543.0s551.7s2,67530,288993,243$2.5855MediumPublishedAdd + persistence passed
Gemini 4 ArgonGemini CLIComplete740.2s771.3s159,59736,4332,056,456$0.8892Native defaultPublishedAdd + persistence passed
Kimi K3.1 ProKimi CodeComplete909.7s936.1s95,84916,561983,808$0.5541LowPublishedAdd + persistence passed
Grok 4.7Cursor CloudComplete1261.7s1273.5sUnknownUnknownUnknownUnknownNative defaultPublishedAdd + persistence passed
GLM 5.3ZCodeComplete1795.2s1849.5s388,479186,69310,088,064$1.4699Native defaultPublishedAdd + persistence passed
Download results JSON ↗

Sorted by time taken, fastest first.

GPT-5.6 Sol grocery app

GPT-5.6 Sol

140.7s · $0.6407

Add + persistence passed

Agent reported 5 passing Node tests and a Chrome interaction check. Google Fonts has a system-font fallback.

DeepSeek V4.1 Flash grocery app

DeepSeek V4.1 Flash

152.4s · $0.0482

Add + persistence passed

Agent reported 37 passing unit/DOM tests, syntax checks and a successful build. The DOM tests used happy-dom, not a real browser.

GPT-6.1 Sol grocery app

GPT-6.1 Sol

244.4s · $0.2618

Add + persistence passed

Agent reported 27 passing browser checks and syntax checks.

Muse Wonder grocery app

Muse Wonder

256.9s · $1.2537

Add + persistence passed

Agent reported 19 passing DOM-harness tests and syntax checks; no real-browser test. Usage includes Spark helpers.

GPT-6 Astra grocery app

GPT-6 Astra

275.7s · $1.4420

Add + persistence passed

Agent reported 6 passing Chromium tests and syntax checks.

Claude Opus 5.5 grocery app

Claude Opus 5.5

451.1s · $1.3333

Add + persistence passed

Agent reported 8 passing unit tests, 12 passing browser tests and a successful typecheck/build. Recorded usage includes Claude Haiku 5.5.

Devin Normal grocery app

Devin Normal

451.6s · Unknown

Add + persistence passed

Agent reported 9 passing unit tests, build/lint and 15 passing browser tests. Devin uses its own VM and agent mode. Token usage is unavailable.

Claude Fable 5.1 grocery app

Claude Fable 5.1

543.0s · $2.5855

Add + persistence passed

Agent reported 6 passing Playwright tests and a successful typecheck/build. Recorded usage also includes claude-haiku-5-5.

Gemini 4 Argon grocery app

Gemini 4 Argon

740.2s · $0.8892

Add + persistence passed

Agent reported 6 passing unit/integration tests and a successful typecheck/build. Native reasoning default.

Kimi K3.1 Pro grocery app

Kimi K3.1 Pro

909.7s · $0.5541

Add + persistence passed

Agent reported 21 passing Chrome checks and syntax checks. Kimi used low effort. Optional Google Fonts has a system fallback.

Grok 4.7 grocery app

Grok 4.7

1261.7s · Unknown

Add + persistence passed

Agent reported 10 passing unit tests, typecheck/build and browser checks. Cursor native token cost is unavailable.

GLM 5.3 grocery app

GLM 5.3

1795.2s · $1.4699

Add + persistence passed

Agent reported completed app checks. Native reasoning default; generic Artos effort does not control ZCode. The agent used a visual-judge subagent despite the task's no-delegation instruction.