Can Jev or Gemini pick the right Help Center article?

A retrieval experiment on the public Netflix Help Center, with the same two setups run on two models

11 November 2026 · English, signed-out, 1,600-article snapshot, 33 test questions · jev-1.14.0 and gemini-decision-1

The question

I wanted to test whether decision models can locate the right Help Center article better than existing solutions such as the Netflix Help Center search API. Two such models are compared: Jev (jev-1.14.0) from TypeSafe and Gemini (gemini-decision-1) from Google. Both do the same job: they are given a list of options and return a probability for each. Each is run in the same two setups.

The five systems compared:

  1. The Netflix Help Center search API.
  2. Jev 255×7.
  3. Jev topic-first.
  4. Gemini 255×7.
  5. Gemini topic-first.

What I did

  1. I used the same 33 questions (24 Normal, 9 Hard) for every system. The four decision-model systems used the same English (en) Help Center snapshot (2026-09-18). The Netflix Search API baseline was queried separately against the live, signed-out service on 6 November 2026 UTC, so its candidate corpus may have differed from the saved snapshot. A GPT flagship model wrote the questions and drafted the gold labels; I approved them.
  2. I gave both models identical setups: the same groups, the same option text, the same instructions and the same call structure. Only the model behind the call differs.
  3. I summarize the results numerically on the Results tab and walk through individual questions in a detailed case study on the Cases tab.

Disclaimer

This is a direction, not a verdict. The questions were written for this test, not taken from real customers. With only nine Hard questions, one case moves a Hard score by about eleven points. The gold labels were drafted with AI help and approved by one person (me). Each system was run once for the headline numbers, so a difference of one or two questions should not be read as a real gap. Gemini was run on a free-tier key limited to 50,000 input tokens per minute, which affects its time per question but not its answers.

Not compared: the Netflix chatbot (its answers were not repeatable between chat sessions), Games questions (many near-identical articles), signed-in customers, other languages, and real traffic.

Title-overlap sensitivity

Some questions repeat much of the expected article title. The table below applies the benchmark's current guard—more than 60% of the primary article title's non-stopword tokens appear in the question—and recalculates the headline metric without those questions. This is a sensitivity check, not a replacement score: real customers can enter exact titles and error codes, but title-heavy synthetic questions can make absolute retrieval results look stronger.

Strict Hit@1 with and without title-heavy questions
System All 33 Excluding 8 title-heavy questions
Netflix Search API 70% (23/33) 60% (15/25)
Jev 255×7 88% (29/33) 88% (22/25)
Jev topic-first 85% (28/33) 80% (20/25)
Gemini 255×7 88% (29/33) 88% (22/25)
Gemini topic-first 88% (29/33) 84% (21/25)

Excluded by this check: cannot-sign-in (67%), lost-email-phone (71%), payment-options (100%), plans-pricing (100%), black-screen-sound (67%), network-connection (100%), download-failed (75%), household-error (63%). The percentage is the share of the expected article title's content tokens repeated in the question.

Tested solutions

System How it works
Netflix Search API Sends the customer question to the public, signed-out GET /en/api/search and keeps the first 5 ranked results.
Jev 255×7 7 groups of up to 255 articles. Each group passes up to 5 top articles Jev rated into 1 final call for Jev to rank. In total, 8 Jev calls per question.
Jev topic-first Jev picks up to 3 of 34 inferred topics (cumulative probability 0.9), then selects among those topics' articles: 1 call when they fit in 255, otherwise groups of up to 255 that pass up to 5 articles each into a final call.
Gemini 255×7 The same 7 groups, prompts and final call as Jev 255×7, answered by Gemini. In total, 8 Gemini calls per question.
Gemini topic-first Gemini picks up to 3 topics (cumulative probability 0.9), then selects among the topics' articles in groups of up to 255; groups pass up to 5 articles into a final call when there is more than one group.

How 255×7 works

One model call can hold at most 255 options, so the 1,600-article corpus does not fit in one call. A tournament gets around that.

  1. Split the corpus into 7 groups of up to 255 articles (six of 255 and one of 70).
  2. Run 7 calls in parallel, one per group. Each call returns a probability for every article in its group.
  3. From each group keep up to 5 articles whose probability is above zero, the winner first. Ties go to the alphabetically first key.
  4. Run one final call over the carried articles (up to 35). Its probabilities give the final ranking and the top 5.

Why a final call? A group's winner is not necessarily among the overall top 5. Each group's probabilities add up to 1 even when nothing in the group fits (Observed in both vendors' documentation), so a weak group's best article can look as certain as a strong group's. Probabilities from different groups cannot be compared directly, so the carried articles go through one more call that scores them together.

That is 8 calls per question. I keep the top 5 of each group, not only the winner, so that the final top 5 can contain more relevant articles. Passing only each group's winner is run as a comparison on the Results tab.

255×7 tournament: seven group calls each carry up to five articles into a final call whose probabilities give the ranking
255×7 tournament: seven group calls each carry up to five articles into a final call whose probabilities give the ranking

How topic-first works

Topic-first first asks the model which topic the question belongs to, and only then searches the articles in that topic. It needs a topic list. I do not have Netflix's, so I inferred 34 topics from article titles and breadcrumbs (full list in the Appendix). As a spot check, a separate AI review agreed with the assigned topic on 60 of 66 sampled articles (90.9%; up to 2 per topic). Each article has exactly one topic.

Step by step

  1. Router call. One call over the 34 topics. Each option is a topic described by its name, its broader category and up to five example article titles. The model returns a probability for every topic.
  2. Keep up to 3 topics. Take topics from the most probable down, and stop as soon as their probabilities add up to 0.9 or more, or when 3 topics are kept. A confident router keeps one topic; an unsure router keeps up to three.
  3. Pool the candidates. The candidates are all articles in the kept topics.
  4. Search the pool.
    • 255 candidates or fewer: one group call over the pool. Its probabilities give the ranking, and there is no final call.
    • More than 255: split the pool into balanced groups of at most 255, carry the top 5 of each group, and run one final call over the carried articles. With 256 to 511 candidates that is 2 groups, up to 10 carried articles and one final call.
    • One candidate: no call is needed.
  5. Calls per question. 2 when the pool fits one group (router plus one group call), 4 for two groups (router, two groups, final call), 5 for three groups.
Topic-first: a router picks up to three topics, the pooled articles go to one group call or to groups plus a final call
Topic-first: a router picks up to three topics, the pooled articles go to one group call or to groups plus a final call

The beam is my code, not a model feature. Neither Jev nor Gemini offers a "keep several topics" option. Both return a probability for every choice, and those probabilities sum to 1 (Observed in every logged router and group call of both models). So at most one topic can be above 0.9, and the 0.9 is a cumulative threshold, not a per-topic one. This repository's own code adds the topics' probabilities, highest first, until the running total reaches 0.9 or 3 topics are kept. Observed: Gemini's router kept 1 topic for 25 of 33 questions, against 26 for Jev, because the top topic alone already reached 0.9.

Two Observed examples from Gemini's router, with the running total after each topic:

  • 2 topics. "There is a Netflix charge on my card that I do not recognize." account_security 0.743, then unexpected_charge 0.250 gives a total of 0.992, so the beam stops at 2.
  • 3 topics. "The Netflix picture is too dark to see clearly." audio_video 0.660, buffering_slow 0.203 (total 0.862), then other_support_issue 0.084 gives 0.946, so the beam stops at 3. The limit of 3 would apply even if the total were still below 0.9.
Topic-first, measured over the 33 gold questions
Measure Jev topic-first Gemini topic-first
Questions 33 33
Router kept 1 topic 26 25
Router kept 2 topics 4 2
Router kept 3 topics 3 6
Gold topic among those kept 32/33 33/33

A one-topic variant

I also tested keeping only the router's top topic. It was not run fresh: it is derived from the up-to-3 data, keeping only the top topic's articles from the saved group-call probabilities and taking the top 5 by probability, with no final call. When the pool already fitted one group, the derivation is exact for what the model returned. When it did not, the up-to-3 ranking is reused. A small live run of the true one-topic setup checks the derivation. The results are on the Results tab under Variants and derived setups: the "1 topic" rows in the table, and the paragraph "Keeping one topic" below it.

What Gemini Decision 1 offers

Observed from Google's public documentation on 11 November 2026 (see the capability record): a Choice question takes 2 to 255 options; the context window is 262,144 tokens; option keys are sorted alphabetically in UTF-8 byte order before evaluation, so request order has no effect and ties go to the alphabetically first key; questions in one request are independent; price is $0.10 per million input tokens and output is not charged. Jev's Choice takes 1 to 255 options, and its default context is 64k tokens per request, at $0.042 per million input tokens.

Observed in live probes (one request per check, no question or article text kept):

  • The 255-option cap is enforced. 255 options returned HTTP 200 and 256 returned HTTP 400 with the message "must have between 2 and 255 options". An 800-option Choice is therefore not possible on either model, and 255×7 is the largest single-stage split.
  • The key worked as an ordinary API key. There was no early-access gate. It is a free-tier key: the service answered HTTP 429 with "limit: 50000 input tokens per minute on Free Tier".
  • Probabilities have full precision. A 255-option call returned up to 22 decimal places and no option was exactly zero. Jev rounds to two decimals, so most of its options are exactly zero. For Gemini, "keep up to 5 articles with probability above zero" always keeps 5, so the carry rule is effectively "top 5 by probability".
  • The response does not name the model. The report records the requested model ID only.
  • Identical requests gave identical probabilities (three repeats, largest difference 0).

Inferred Gemini's alphabetical sort puts article-10 before article-9, so option order inside a group differs from the one Jev sees. I did not test whether order matters on either model.

Live probe of gemini-decision-1
Response fields answers (type, choice, probabilities, confidence) and usage; does not name the model
255 options HTTP 200
256 options HTTP 400: CreateDecisionRequest.questions[choice].choice.criteria: must have between 2 and 255 options
Probability precision up to 22 decimals; 0 of 255 options exactly zero
Three identical requests identical probabilities (largest difference 0); same choice each time
Request latency, 255 options 691, 573, 635 ms for 13436 input tokens
Retries on those requests 0, 0, 1

Summary

All questions (33)

System Strict Hit@1 Lenient Hit@1 Recall@5 MRR
Netflix Search API 70% 70% 76% 72%
Jev 255×7 88% 🥇 94% 🥇 97% 🥇 95% 🥈
Jev topic-first 85% 88% 94% 91%
Gemini 255×7 88% 🥇 94% 🥇 97% 🥇 95% 🥇
Gemini topic-first 88% 🥇 91% 🥉 97% 🥇 93% 🥉
Strict Hit@1
Share of questions where the first result is the gold article. This is the headline number.
Lenient Hit@1
Same, but the first result may also be any acceptable alternative the gold list names.
Recall@5
Share of questions where the gold article or an acceptable alternative appears anywhere in the top five.
MRR
Mean reciprocal rank: 1 if the first match (gold or acceptable alternative) is at rank 1, 1/2 at rank 2, down to 1/5 at rank 5, and 0 if none of the top five match. Rewards ranking a match high.
Medals
🥇 🥈 🥉 mark the top three systems in each column for the selected questions. Tied systems share a medal.

Jev returns one answer. For Jev, Recall@5 and MRR rank the other champions by their final-round probabilities, so that Jev can be compared with a five-result search.

What the numbers say

Both Jev setups beat the Netflix API on this question set. Observed The API ranked the right article first for 88% (21/24) of the Normal questions and 22% (2/9) of the Hard ones. Jev 255×7 reached 96% (23/24) and 67% (6/9); Jev topic-first reached 96% (23/24) and 56% (5/9). Confidence: high for this set, low for real traffic. Other explanation: the questions are AI-written and may suit an AI reader.

Jev 255×7 and topic-first are close. Observed They differ by one question overall. Topic-first needs 2.2 calls per question on average, against 8 for 255×7.

Gemini and Jev are about tied on this set. Observed Gemini 255×7 scored 96% (23/24) on Normal and 67% (6/9) on Hard; Gemini topic-first scored 96% (23/24) and 67% (6/9). Overall, Gemini 255×7 and Jev 255×7 ranked the right article first for the same 29 of 33 questions. Gemini topic-first got 29 against Jev's 28, a difference of one question. Confidence: medium that they are about equal here, low for any ranking. Other explanation: with 33 questions, a difference of one question is inside the noise.

The router behaves differently. Observed Gemini's router kept three topics for 6 questions against 3 for Jev, and its kept topics contained the right one in every question. Inferred: Gemini's unrounded probabilities are less peaked at the top, so the 0.9 threshold takes more topics. Other explanation: the two models simply disagree on a few borderline topics.

Variants and derived setups

Setup variants (strict Hit@1 unless stated)
Setup Where the numbers come from All Normal Hard Recall@5 MRR
Jev 255×7 Live run (headline) 88% 96% 67% 97% 95%
Jev 255×7, winner only Live run (same group calls) 91% 100% 67% 97% 93%
Jev topic-first Live run (headline) 85% 96% 56% 94% 91%
Jev topic-first, 1 topic Derived from logged calls 85% 96% 56% 91% 89%
Gemini 255×7 Live run (headline) 88% 96% 67% 97% 95%
Gemini 255×7, winner only Live run (same group calls) 88% 96% 67% 97% 92%
Gemini topic-first Live run (headline) 88% 96% 67% 97% 93%
Gemini topic-first, 1 topic Derived from logged calls 88% 96% 67% 94% 92%

Derived rows re-rank the probabilities saved from the live calls. They make no new model calls.

Passing the top 5 of each group, not only the winner. Observed In the variants table, Hit@1 with top-5 carry was the same as winner-only on Gemini (88%) and one question lower on Jev (88% against 91%). Recall@5 was the same (97%), and MRR was higher with top-5 carry (Jev 95% against 93%, Gemini 95% against 92%). So carrying 5 did not improve the first-place answer here, and it ordered the top 5 slightly better. Confidence: low. Other explanation: a larger test set might show no effect.

Keeping one topic is slightly worse than keeping up to 3. Observed Hit@1 was the same on both models, but Recall@5 was one question lower on each (Jev 94% to 91%, Gemini 97% to 94%) and MRR fell a little (Jev 91% to 89%, Gemini 93% to 92%), as the variants table above shows. The lost question is the same on both models. Confidence: low, since one question is inside the noise on a 33-question set, and these rows are derived rather than run fresh. A live check of the true one-topic setup agreed with the derivation on the top choice for 5 of 7 questions on Jev and 6 of 8 on Gemini. Inferred: the gaps come from the live router and group calls differing slightly from the saved ones, and from the derivation ranking within a larger group's probabilities rather than over the one topic alone. Confidence for the live check: low, seven or eight questions.

Model cost and latency

Per question. Latency counts each request once, without rate-limit waits and retries.
System Model calls Input tokens Median latency Model cost
Netflix Search API — — 226 ms —
Jev 255×7 8 108k 420 ms $0.0045
Jev topic-first 2.2 8k 172 ms $0.0003
Gemini 255×7 8 87k 1512 ms $0.0087
Gemini topic-first 2.1 7k 552 ms $0.0007

Model cost is worked out from tokens, not taken from a bill. It covers the model calls only, not hosting, indexing or engineering time. Each response reports its input tokens; I averaged them per question and multiplied by the listed price: $0.042 per million input tokens for Jev and $0.10 for Gemini. Output tokens are not charged by either. The Netflix API has no model cost.

Gemini's model cost is about twice Jev's per question for the same setup. Observed It is billed at a higher rate per token, and it counted fewer input tokens per 255×7 question than Jev (about 87k against 108k for the same text), so the cost gap is smaller than the price gap.

How to read latency. Observed Timings come from one local machine over the public internet, not a production system, so treat them as indicative. The latency column counts each request once (router, then group calls 4 at a time, then the final call) and leaves out rate-limit waits. Gemini's real wall time was much longer, about 120 s per question for 255×7, because of the free-tier limit of 50,000 input tokens per minute: a 255×7 question sends about 87,000, so 155 of 266 group calls had to wait and retry. That is a quota effect, not model speed, and a paid tier was not tested.

Duplicate articles returned

Observed In the snapshot, 17 groups of articles share identical text, covering 72 articles. A further 406 pages are placeholders reading "this information is not available for United States", with different titles.

A pair to look at: Netflix Error 2.2 (125404) and Netflix Error 1.2 (124914) have the same text: "We are currently researching this error to determine the right solution. To help us investigate, contact us." Only the title differs, so only the title can tell a search system which one a customer meant.

Largest group: 7 copies of "What happens to games if I cancel?".

How often each system returned an identical-text copy or a placeholder page (across 33 questions)
System Copy, as first result Copy, in top 5 Placeholder, first result Placeholder, in top 5
Netflix Search API 0 2 0 9
Jev 255×7 0 0 0 1
Jev topic-first 0 0 0 0
Gemini 255×7 0 1 0 1
Gemini topic-first 0 0 0 9

Case results

  • Hard questions are listed first.
  • Open a row to see the original question and each system's top five articles, with the gold answer highlighted.
  • Click any article name to open it on the Netflix Help Center.
Difficulty Case Expected article Netflix Search API Jev 255×7 Jev topic-first Gemini 255×7 Gemini topic-first

Conclusions

Each claim has a confidence level and at least one alternative explanation.

1. Both models can pick the right Help Center article from the saved 1,600-article corpus, and both beat the separately collected live Netflix search baseline on this set. Observed Right article first: API 70% (23/33); Jev 255×7 88% (29/33); Jev topic-first 85% (28/33); Gemini 255×7 88% (29/33); Gemini topic-first 88% (29/33). The model systems used the September snapshot; the API was queried live on 6 November 2026 UTC, so this is not a same-corpus comparison. Confidence: high for the recorded set, low for real traffic. Other explanation: AI-written questions may suit an AI reader, and the API's live corpus may have differed.

2. Gemini and Jev are about tied on accuracy; Jev is cheaper and faster. Observed With 255×7 they ranked the right article first on the same 29 questions; with topic-first they differ by one. Gemini's model cost per question is about twice Jev's, and its requests take longer (about 1.5 s against 0.4 s per 255×7 question, rate-limit waits not counted). Confidence: medium for "about equal". Other explanation: 33 questions cannot separate models this close, and the latency comes from one local machine, not a production system.

3. Topic-first is about as accurate as 255×7 at a fraction of the cost. Observed On both models Hit@1 was within one question of 255×7, with 2.2 (Jev) and 2.1 (Gemini) calls per question instead of 8, about 13 times less input and model cost, and less than half the latency. The router kept the right topic for 32 of 33 questions on Jev and 33 of 33 on Gemini. Confidence: medium. Other explanation: it depends on my inferred topic list; a question whose topic the router drops cannot be recovered, and a real topic tree might route better or worse.

4. Carrying the top 5 of each group did not improve the first-place answer over carrying only the winner, but ordered the top 5 slightly better. Observed Hit@1 was the same or one question lower; MRR was higher on both models. Confidence: low. Other explanation: the differences are one question or a few points of MRR.

5. Keeping only one topic was slightly worse than keeping up to 3. Observed, with a derived setup. Hit@1 was the same; Recall@5 was one question lower on both models. Confidence: low. Other explanation: the up-to-3 beam may matter more on questions this set does not contain.

Next steps

  1. Repeat each system. I did not repeat the Gemini 255×7 run, because on a free-tier key it takes about an hour. Repeats would measure noise before any one-question difference is read.
  2. Test a paid tier for Gemini. The free-tier limit of 50,000 input tokens per minute dominates time per question for 255×7. A paid tier's limits and latency under load were not tested.
  3. Build the test set from real queries. Use top queries from Search Console and internal search logs, including zero-result and rephrased searches, with personal data removed.
  4. Have support experts review the labels, with a second reviewer on every Hard question.
  5. Try topic-first on Netflix's real topic tree. Mine is inferred from titles.
  6. Run an online test comparing deflection and contact rate against current search, with a latency budget, version pinning and monitoring of low-confidence answers.

Versions

Gemini Decision 1

Model requested gemini-decision-1
Model ID recorded gemini-decision-1 (responses do not name the model, so this is the requested ID)
Endpoint POST https://generativelanguage.googleapis.com/v1beta/decisions
Client Direct REST calls from this repository (the AI SDK could not be installed from the available registry).
Listed price $0.10 per million input tokens; output is not charged.

Jev

Model requested jev-1.14.0
Model reported by responses jev-1.14.0

Netflix Search API

Request GET https://help.netflix.com/en/api/search with query=<question>, page=1, perPage=5
Scope en, signed out
API version Not exposed by the service. Identified by its URL, parameters and response fields.
Collected 2026-11-06 00:31:59 to 00:32:34 UTC

AI models

The AI models that helped build the experiment and this report.

Role Model Used for
Question and answer drafting GPT-5.5 Sol Writing the test questions and picking the best-matching article for each.
Coding and drafting assistant Claude Sonnet 5.5 Cursor agent mode Running the experiments, writing the code, reviewing the questions, and building the reports.

Topic list

The topics the router chooses from in topic-first, used in How topic-first works.

The 34 topics the router chooses from (1600 articles). Select a column heading to sort.
Unclassified Netflix error-code troubleshooting 414
Netflix game installation, progress, or gameplay problem 294
Billing through a partner or package 228
Install, update, or set up Netflix 227
Other product or support problem 167
Cannot connect or network error 60
Playback error or title will not play 34
Download unavailable, failed, or expired 29
Create, switch, transfer, or edit a profile 16
Accessibility feature unavailable or difficult to use 14
Audio, picture, or synchronization problem 11
Add, update, or remove payment method 10
Password reset or rejected password 8
Subtitles, captions, or language problem 8
Cannot sign in 7
Parental controls, maturity ratings, or profile PIN 7
Account email or phone access 6
Netflix Household or extra-member problem 6
Search, browsing, or recommendations problem 6
Gift card redemption or balance 5
Playback controls, notifications, or app settings 5
Title, episode, or season missing or unavailable 5
Account compromised or unrecognized access 4
Ads unavailable, excessive, or not working 4
App freezes, crashes, or stops responding 4
Buffering, slow loading, or low quality 3
Device or app is unsupported or incompatible 3
Payment failed or account on hold 3
Privacy, data access, or deletion request 3
Black or blank screen 2
Cancel membership 2
Plans, pricing, or plan changes 2
Unexpected, duplicate, or post-cancellation charge 2
Sign out, deactivate, or remove a device 1

Each article has one topic. A topic above 255 articles cannot fit in one call and is split into balanced groups.