Compare the architectures.
Check what the extra steps deliver.
I built Python pipelines to turn technical support records into knowledge articles,
then compared single-pass prompts with an agent review loop. The work connects model
orchestration, recoverable batch runs and evaluation.
Useful documentation needs
the details to survive.
Resolved software issues combine discussion, diagnostic logs, configuration and code
changes. Turning that material into a knowledge article is useful operational work,
but a fluent summary can still omit the fix or invent an explanation.
Build.
I implemented configurable Python generation and evaluation pipelines, including a
LangGraph generator–auditor loop and a local Ollama backend.
Compare.
I tested simpler single-pass prompts alongside the agent workflow, using the same
selected technical records within each evaluation.
Inspect.
I retained outputs, scores and run records to investigate where added review helped,
where it did not, and what the measurements could support.
An offline research project based on 69 purposively selected GitHub issue/PR records:
40 from Kubernetes, 20 from Ansible and 9 from VS Code.
02 / Implemented architecture
One task.
Several ways to generate and check.
I separated generation, review and evaluation so each stage could be configured and
inspected. The finalisation step selects and saves an existing draft.
Generate, audit, then decide
Two LLM roles and a deterministic finalisation node.
Implemented
Selected inputIssue context + PR diff
Context limits applied by the runner
↓
LLM role 1Generator
Draft a Markdown knowledge article
↓
LLM role 2Auditor
Return scores, a decision and feedback
ReviseFeedback → Generator
Retry the draft while the review-iteration budget remains.
FinishSelect → save
Accept a passing draft, or select the highest-scored draft at the iteration
limit and record the fallback.
LangGraph carries the current draft, feedback, iteration count and best-scored
draft. Finalisation is deterministic. The review budget limits draft iterations;
network retries have separate logic.
A simpler comparison
Keep the generation context together and vary the prompt.
Implemented
Selected inputIssue context + PR diff
Same underlying task records
↓
BaselineDirect generation
Request the knowledge article in one model call.
CoT variantReasoning prompt
Add an explicit Chain-of-Thought instruction before the article.
A further variant adds the review loop to CoT generation. Run 09 compares these
two CoT workflows; the evaluation notes describe how each path saved its output.
Compare saved outputs
Generation and scoring are separate batch steps.
Saved results
Matched recordsCandidate articles + reference material
Paired by the original issue/PR identifier
↓
LLM-as-a-JudgeFactuality & technical completeness
Structured scores and an explanation
Reference material includes the PR description and diff. Generation also receives
the diff. The judge scores saved candidate articles using a 0–100 rubric.
Interchangeable model providers
A provider factory supports cloud APIs and local Ollama. I used Gemini and local
Llama 3 in the experiments, keeping provider selection separate from the workflow.
Recoverable batch work
Configuration, cached intermediate files, saved results and failure records let me
inspect interrupted batches and reuse completed work. Cache versioning and retry
behaviour are concrete areas for the next iteration.
03 / Decisions & trade-offs
Make complexity earn its place.
01
Keep cloud and local inference interchangeable
The model interface allowed experimentation to move from Gemini API calls to local
Llama 3 through Ollama. This made local inference an option when API limits and
running costs affected the experiments.
Trade-off
Local inference trades API access and usage limits for local hardware capacity,
runtime and smaller-model constraints.
02
Give the review loop an exit
The workflow tracks the best-scored draft, stops on an auditor pass and uses a
recorded fallback when it reaches the configured iteration limit. Batch outputs
make these outcomes inspectable.
Trade-off
A completed run may still be an auditor-rejected fallback. Completion, review
acceptance and factual correctness are separate signals.
03
Keep simpler baselines in the comparison
I began by exploring multi-agent review, then added single-pass CoT as a simpler
alternative. Comparing both made the extra review stage a testable choice.
Trade-off
A model judge makes batch comparison practical. Its rubric and presentation
settings also shape the scores, so human evaluation is a useful next step.
04 / Results & interpretation
Single-pass CoT led Run 08.
Run 09 left the review effect uncertain.
Single-pass CoT received the highest average score in the retained three-way
comparison. Adding a same-model reviewer in a separate experiment showed no
statistically significant benefit.
The figures show the configured LLM judge's 0–100 scores for 69 paired records. Stored
scores were recalculated on 2 October 2026; this was an archive review, without a new
model run.
Run 08 · Three-way evaluation.
Llama 3 generation and judging as recorded for this experiment. Points show means
for the same 69 records within this run; error bars show sample standard deviation.
Run 08: LLM-judge scores on a 0–100 scale
Workflow
Mean
Sample SD
Single pass
79.88
10.25
Multi-agent
76.74
14.60
Single-pass CoT
91.01
6.45
CoT scored 14.28 points above the multi-agent workflow in this run (paired t-test, p ≈
1.65 × 10
−13
). For these records and this judge configuration, the simpler CoT workflow was the
stronger baseline.
Run 09 · Reviewer-loop comparison.
A separate pairwise evaluation of CoT variants, with 69 matched records. Run 08 used
three-way judging; each run is interpreted within its own evaluation setting.
The reviewed outputs averaged 0.90 points lower (95% confidence interval for the
paired difference: −4.53 to +2.73; p = 0.6232). This result is statistically
inconclusive: the interval spans both lower and higher scores. It supports keeping
single-pass CoT as the baseline while testing changes to the review stage.
Evaluation method and follow-up checks
These are LLM-judge scores on a 0–100 rubric, rather than measured accuracy
percentages. The 69-record sample was purposive and unevenly distributed across
three open-source projects. Paired t-tests are exploratory, with unadjusted
p-values.
Candidate method names and ordering were visible to the judge. Randomised, blinded
presentation and human review would help assess ordering, style and
self-preference effects.
Single-pass CoT saved the full response. The multi-agent CoT path removed
reasoning text on fresh generation but retained it on cache hits. A follow-up
comparison should normalise this handling across paths.
Cache keys captured only part of the model, prompt and input configuration. A
versioned experiment manifest should also capture the model digest, seed and
environment for a reproducible rerun.
Both generation and evaluation receive PR diffs. The comparison measures article
quality with access to the fix; a separate experiment would be needed to isolate a
mechanism such as information loss during review.
Timing records include possible retry waits. A runtime comparison would need
consistent hardware and warm-up conditions.
05 / Illustrative example
From a resolved issue
to a usable knowledge note.
This small example shows the intended transformation and the details an evaluator
would need to check.
Handwritten, synthetic example.
This illustration shows the target article structure for the case study.
Example input
A worker skips its final batch
A batch worker receives 23 records with a batch size of 10. It processes 20
records and leaves three behind without reporting an error.
The loop calculated its batch count using integer division. The fix iterates over
record offsets, including the final partial batch.
for start in range(0, len(records), batch_size):
process(records[start:start + batch_size])
The proposed regression check uses 23 records and checks that every record is
processed once.
Example knowledge note
Preserve the partial batch
Symptom
The final three records remain unprocessed when 23 records are handled in
batches of 10.
Cause
Integer division counts only complete batches.
Resolution
Iterate from zero to the record count in batch-size steps. Each slice includes
the available records, including a shorter final slice.
Verification
Check a non-multiple batch size and assert that all input records are
processed exactly once.
The useful details are the boundary condition, the actual fix and a regression check
that targets the failure. A review should preserve all three.
06 / Work & evidence
Inspect the workflow.
Recalculate the comparison.
Implementation and experiment records
Area
Work completed
What to inspect
Generation & orchestration
Implemented
Python runners, two-role LangGraph review loop, model adapters, caching and
saved fallback outcomes.
Offline experiments
Scores recalculated
Stored output and evaluation records, with a score export and script for
recalculating Run 08 and Run 09 statistics.
Workflow diagnostics
Inspection scripts
Scripts print workflow behaviour and outcomes for manual inspection during
development.
The public export contains scores and pseudonymous case labels. The script
recalculates its means, sample variation, paired t-tests and confidence intervals.
Original records and research archives remain private.
The next iteration would combine consistent output handling and versioned experiment
manifests with human evaluation, tool-assisted review and broader model comparisons.
The aim is to identify which review steps improve the article enough to justify their
added complexity.
