Harness benchmark / 2026-11-04

GPT-5.6 Terra Custom skills supplied: None

Main prompt

Build a polished, responsive grocery list web app. Work only in /home/user/grocery-trial (or the backend project directory if that absolute path is unavailable). The sole starting file is README.md with these exact contents:
# Grocery list trial

Starting point: empty app. Implement according to the trial prompt.

If the backend does not have that file, create it verbatim before implementing. Do not use unrelated existing project files.
Requirements: add grocery items with name, quantity and category; edit and delete items; mark purchased/unpurchased; filter all/remaining/purchased; search; clear purchased; show remaining count; persist everything in localStorage. Include useful initial sample groceries only on first load. Make an attractive, accessible mobile and desktop UI with keyboard-friendly controls and empty states. Use a simple self-contained frontend stack of your choice, no external paid services or API keys. Test core flows and run build/tests as appropriate. Do not delegate to other agents or invoke additional AI.
Launch an app preview as a detached process (nohup setsid), listening on 0.0.0.0. On Boat use host <port> --public, adding --localhost only if loopback-bound. Verify the public token-free HTTPS URL responds. On hosted backends use their available preview mechanism; if Boat hosting is unavailable explain this and return the native public preview if possible. Never expose unrelated services.
Return a concise summary with implemented features, tests and their outcomes, preview URL, project path, and limitations. Do not stop the preview server.

Time taken

Execution time, seconds

  1. Codex49.9s
  2. OpenCode59.5s
  3. Hermes80.8s
  4. Prime Agent84.9s
  5. Pi94.9s
  6. JetBrains Junie100.4s
  7. Cursor Cloud153.4s
  8. Factory Droid168.4s

Cost

Priced token subtotal, USD · * incomplete · — unknown

  1. Codex$0.1094
  2. Hermes$0.1647*
  3. OpenCode$0.1734
  4. Prime Agent$0.2102
  5. JetBrains Junie$0.2838
  6. Pi$0.3069*
  7. Factory Droid$0.4742*
  8. Cursor Cloud—
Grocery app benchmark results
HarnessRunTotalInput tokensOutput tokensCache readReasoningOriginal previewPublication check
CodexComplete49.9s59.4s16,8205,41553,737$0.1094MediumPublishedAdd + persistence passed
OpenCodeComplete59.5s188.6s29,5785,949224,185$0.1734Not exposedPublishing failedAdd + persistence passed
HermesComplete80.8s234.1s25,0535,735228,921$0.1647*Not exposedPublishedStartup error
Prime AgentComplete84.9s280.7s35,6566,788287,166$0.2102MediumPublishedAdd + persistence passed
PiComplete94.9s105.2s57,5948,294460,836$0.3069*MediumPublishedAdd + persistence passed
JetBrains JunieComplete100.4s124.8s49,02610,532296,871$0.2838Not exposedPublishing failedAdd + persistence passed
Cursor CloudComplete153.4s216.0sUnknownUnknownUnknownUnknownNot exposedNo public URLAdd + persistence passed
Factory DroidComplete168.4s184.9s37,09726,000439,953$0.4742*Not exposedPublishing failedAdd + persistence passed
Download results JSON ↗

Sorted by time taken, fastest first.

Codex grocery app

Codex

49.9s · $0.1094

Add + persistence passed

Agent reported 2 passing tests.

OpenCode grocery app

OpenCode

59.5s · $0.1734

Add + persistence passed

Agent reported 2 passing tests. Usage includes GPT-5.4 Nano: 1,037 input and 13 output tokens. Public preview needed separate recovery.

Hermes grocery app

Hermes

80.8s · $0.1647*

Startup error

Agent reported static core-flow and HTTP checks. Publication check: fresh storage triggers “Cannot access save before initialization”. Original app preserved.

Prime Agent grocery app

Prime Agent

84.9s · $0.2102

Add + persistence passed

Agent reported syntax and local checks. Usage was provisional in the first summary and has since settled.

Pi grocery app

Pi

94.9s · $0.3069*

Add + persistence passed

Agent reported 3 passing tests. Editing uses native prompt dialogs.

JetBrains Junie grocery app

JetBrains Junie

100.4s · $0.2838

Add + persistence passed

Agent reported 4 passing tests. Public preview needed separate recovery.

Cursor Cloud grocery app

Cursor Cloud

153.4s · Unknown

Add + persistence passed

Agent reported syntax and local HTTP checks. Used Cursor’s native cloud environment. Usage returned zeros and is treated as unknown. Public preview needed separate recovery.

Factory Droid grocery app

Factory Droid

168.4s · $0.4742*

Add + persistence passed

Agent reported syntax and local HTTP checks. Public preview needed separate recovery.