# Mosthofa Imran, full text mirror
Every paper and implementation note as plain text, front matter preserved, so confidence and
state travel with the prose. Generated at build. Canonical HTML at https://mosthofaimran.com/.
If you quote a claim, carry its confidence value with it.
================================================================
SECTION 3: IMPLEMENTATION NOTES
================================================================
--- 3.1 Mevrik: agentic customer experience platform ---
url: https://mosthofaimran.com/impl/mevrik-cx/
state: production
stack: Rust, Python, ClickHouse, PostgreSQL, RabbitMQ, Redis, Kubernetes
result: 40M events/day, 3 regulated tenants
This implementation note is **listed but not written**. The index carries its summary, its
stack and the numbers it produced. The constraints, decisions, measurements and named
failure modes are not yet migrated.
Principle 4.8 says an architecture is not presented without its failure mode, so this page
does not present one until it can.
--- 3.2 Sovereign LLM gateway ---
url: https://mosthofaimran.com/impl/llm-gateway/
state: production
stack: Python (FastAPI), Redis, OpenSearch, PostgreSQL, Ollama
result: 3.1M calls/day, p99 overhead 180 ms
falls over at: ~11k rps. The Redis quota path saturates first and degrades to fail-open with an alarm.
A bank wants agentic customer service. Its regulator wants every token to stay inside the
bank. Those two sentences are the entire project, and everything difficult about it follows
from declining to compromise on either.
## 3. Decisions worth defending
**3.1. Redaction before routing.** Personal data detection runs in process before backend
selection. It costs 14 ms at p99, and it means a misconfigured route cannot leak. A control
that depends on the correctness of the next hop is not a control, it is a hope with a runbook.
**3.2. Deterministic, stamped fallback.** When a backend degrades, the gateway fails over to
a smaller local model and marks the response as having come from the understudy. Downstream
systems can see it, dashboards can count it, and the tenant can decide what it means. A
silent quality drop is worse than an error, because nobody investigates it and the damage
shows up in a churn report six weeks later.
**3.3. Quotas as a first class object.** Per tenant, per model, per minute, sliding window in
Redis. This is not a cost control. It exists because one tenant's retry storm used to be
everybody's outage, which is Principle 4.3 learned the expensive way.
## 6. What I would do differently
Model the audit schema first. We built four adapters and then discovered that a uniform audit
record was the actual product, because it was the thing the tenant's compliance team read and
the only artifact that survived a backend swap.
--- 3.3 Ingest path rewrite, PHP to Rust ---
url: https://mosthofaimran.com/impl/ingest-rs/
state: production
stack: Rust (tokio), RabbitMQ, PostgreSQL, MinIO
result: 89% fewer nodes, p99 340 ms to 11 ms
This implementation note is **listed but not written**. The index carries its summary, its
stack and the numbers it produced. The constraints, decisions, measurements and named
failure modes are not yet migrated.
Principle 4.8 says an architecture is not presented without its failure mode, so this page
does not present one until it can.
--- 3.4 Analytics migration to ClickHouse ---
url: https://mosthofaimran.com/impl/olap-migration/
state: complete
stack: ClickHouse, Kafka Connect, Airflow
result: 14B rows, query p95 9.4 s to 380 ms
This implementation note is **listed but not written**. The index carries its summary, its
stack and the numbers it produced. The constraints, decisions, measurements and named
failure modes are not yet migrated.
Principle 4.8 says an architecture is not presented without its failure mode, so this page
does not present one until it can.
--- 3.5 Air-gapped delivery pipeline ---
url: https://mosthofaimran.com/impl/airgap-delivery/
state: production
stack: OCI bundles, cosign, Helm, offline registry
result: 6 sites, 0 failed installs since 2024-09
This implementation note is **listed but not written**. The index carries its summary, its
stack and the numbers it produced. The constraints, decisions, measurements and named
failure modes are not yet migrated.
Principle 4.8 says an architecture is not presented without its failure mode, so this page
does not present one until it can.
================================================================
SECTION 5: POSITION PAPERS
================================================================
--- 5.1 Competence Porn ---
url: https://mosthofaimran.com/papers/competence-porn/
state: holding
confidence: 0.8
revised: 2026-08-14
retires:
- A longitudinal study showing heavy consumers of technical content outperform matched peers on blind, time-boxed debugging tasks.
- Evidence the effect is generational rather than structural, appearing at equal rate in cohorts who entered the field before ranked feeds existed.
- A large publisher of technical content disclosing what fraction of the architectures it demonstrated reached production and survived twelve months, where that fraction is high.
Abstract. The feedback loop that once rewarded building has been
rerouted to reward the performance of building. The performance is cheaper to produce,
faster to distribute, and structurally unfalsifiable. This paper states the mechanism,
gives its strongest counter-argument the floor, and lists the evidence that would retire
it. Confidence 0.80. The gap from 0.95 is Section 6.
## 1. The claim
**A pilot logs hours. A surgeon logs procedures. We log tabs.**
Somewhere in the last decade this industry discovered that watching a competent person
work is more pleasurable than being one, and, more importantly, that from the inside the
two feel nearly identical. A twelve minute video of someone untangling a difficult module
ends in the same warm, settled feeling as having untangled it yourself. That is not a
defect in you. It is the business model of the medium, and it works because it is pointed
at something real: the pleasure of watching craft is one of the oldest pleasures there is.
Notice what a demo is engineered to remove. There is no data migration. There is no
colleague who left in 2019 carrying the only working mental model of the billing service.
There is no compliance officer, no partial failure, no clock skew, no forty page
procurement questionnaire asking whether the vector store is FIPS validated. A demo is a
jet engine bench tested at sea level and sold as a mountain crossing. Everyone involved
knows this. Nobody is lying. The removal is what makes it watchable.
## 2. Why the numbers look fine
We have more available knowledge per practitioner than at any point in this industry's
history, and a persistent, widely reported sense among practitioners that they are behind.
Those two facts are not in tension. They are the same fact. The supply of things that
resemble learning has outgrown the hours in which learning can actually occur, and the
surplus has to go somewhere. It goes into the feeling.
## 3. Who absorbs the cost
Not, mostly, the person watching. This is the part of the argument I had to rewrite after
the correction recorded in the June 2026 revision, and it is where the paper stopped
being about individuals.
The cost lands on whoever inherits the decision. A demonstrated architecture arrives in a
design review carrying the authority of having been seen working, and the evidence for it
is a recording in which the hard parts were removed by construction. The person who
adopts it pays the removed costs later, one at a time, and pays them in an environment
where the demonstration is no longer available to argue with.
what the demo removed who pays it back
data migration -> the team, in month four
partial failure -> whoever is on call
compliance review -> a person you never met
the departed colleague-> everybody, forever
procurement -> the deal, six weeks late
Figure 1. The demonstration is honest about what it shows. The
liability is in what it removed, and the removal is invisible precisely because
it is what made the demonstration watchable.
The reader correction that produced the June revision was this: the effect is strongest
where tooling is locked down, not where it is abundant. An engineer in a bank with a
restricted toolchain watches more demonstrations, not fewer, because watching is the only
form of access available. That inverts the naive version of the mechanism, in which
abundance causes substitution. Constraint causes it too, and by a different route.
## 4. What survives contact with production
The useful test is not whether a technique is good. It is whether the demonstration
carried the information you would need to operate it. Almost none do, and the gap is
systematic rather than accidental.
What a demonstration shows
What operating it requires
The happy path, end to end
The failure paths, which are the majority of the code and none of the runtime
A fresh, empty system
Behaviour at year three, with accumulated data and two migrations behind it
One operator who wrote it
A rotation of people who did not, at 03:00, holding a runbook
A bounded, chosen problem
An unbounded, inherited one with a compliance constraint attached
Success
A named failure point, which Principle 4.7 asks for and demonstrations never give
This is the same structure as 5.2. Assembly produces the knowledge of what a thing does
and not of how it fails; watching produces even less, and produces it with more
confidence, because watching has no compile step to disagree with you.
## 5. The remedy, which is not "log off"
Abstinence advice is both unrealistic and wrong. Watching skilled people work is a
legitimate and efficient way to learn, and the apprenticeship objection in Section 6 is
strong enough that I will not argue otherwise. The remedy is to attach a consequence to
the watching, because the consequence is the only thing the medium removed that you can
put back yourself.
- **Convert one thing per week into a claim with a cost.** Not notes. A change to
something you own, small enough to ship, that can fail in front of somebody.
- **Predict the failure mode before you look it up.** Write one sentence on how the
demonstrated system breaks. Then find a postmortem. The gap between your sentence and
the postmortem is the measurement, and it is the only calibration signal available.
- **Prefer artefacts with the hard parts left in.** Postmortems, migration write-ups,
capacity notes with a stated breaking point. They are less pleasurable, which is the
point: the pleasure was being produced by the removals.
- **Count what you shipped, not what you consumed.** A pilot logs hours because hours
flown is the quantity that predicts competence. The industry has no equivalent, so the
substitute measure is whatever is easiest to count, and what is easiest to count is
consumption.
None of this is a cure and I am not claiming it as one. It is an attempt to reintroduce
the feedback loop with consequences attached, which is the distinction Section 6 says I
cannot yet state cleanly.
## 6. The strongest objection, unanswered
**Apprenticeship.** All pedagogy involves watching. The apprentice watches the master, and
this has worked for several thousand years. I do not yet have a clean line between
apprenticeship and spectatorship, and until I do, this paper is weaker than its prose
sounds. The distinction I am reaching for involves the presence of a feedback loop with
consequences attached, but I cannot yet state it in a way that survives a determined
counterexample. This objection is the entire reason confidence sits at 0.80 rather than
0.90.
## 7. What this paper does not claim
Technical content is not worthless. The sharpest version of the objection above is that
it is the primary way most people learn, and I have no answer to that. The producers are
not dishonest either: the removals are what make the form work, and everybody involved
knows they are there.
Section 4 is an argument, not a measurement, and it should be read as one. The retirement
conditions state the evidence that would end this paper, and the first of them is the
study I would most like somebody else to run.
--- 5.2 Vibe Coding and the IKEA Effect ---
url: https://mosthofaimran.com/papers/vibe-coding/
state: holding
confidence: 0.75
revised: 2026-08-14
retires:
- A blind study in which engineers who assembled a system without reading its generated internals diagnose induced faults in it at the same rate and speed as engineers who wrote the equivalent system by hand.
- Evidence that the confidence gap in Section 3 closes with tooling rather than with reading, for example a generation workflow whose users predict failure modes as accurately as authors do.
- A demonstration that the effect is about ownership rather than comprehension, appearing at equal strength for code the engineer merely selected rather than assembled, which would make this a paper about a different mechanism.
Abstract. Assembling a working system from parts you never read
produces a strong, sincere sense of understanding it. The labour is real, so the
ownership is real, but the labour was integration and the understanding it purchased is
of the seams rather than of the parts. The gap is invisible while the system works and
becomes the whole problem the first time it fails in a way the assembly did not cover.
This is not an argument about who or what wrote the code. It is an argument about what
reading buys and what assembling does not. Confidence 0.75. Section 5 has the
objection I cannot answer: nobody reads their TLS library either.
## 1. The claim
The IKEA effect is the finding that people place higher value on things they assembled
themselves. The interesting part for engineering is not the valuation. It is that the
sensation of competence produced by assembly is nearly indistinguishable, from the
inside, from the sensation produced by comprehension.
Both feel like understanding. Both produce accurate answers to "what does this do". They
diverge on one question, and it is the only question that matters during an incident:
*how does this fail?*
## 2. Three kinds of knowledge
Knowledge
What buys it
When it is needed
What it does
Reading the interface, or watching it run once
Every day. Cheap and sufficient most of the time.
How it does it
Reading the implementation, or reconstructing it from behaviour
During extension and optimisation. Assembly sometimes supplies this by accident.
How it fails
Reading the error paths, or surviving them in production
During incidents, and only then. Assembly never supplies it.
The third row is where the argument lives. Error paths are the least-read code in any
system: they are rarely exercised in development, rarely covered by the happy-path tests
that assembly produces, and rarely present in the example the code was modelled on.
Generated and copied code inherits this bias, because it is trained on and drawn from
code that is itself mostly happy path.
perceived actual
competence competence
|###########| |####|
|###########| |####|
assembly done what it does
working system how it fails: 0
Figure 1. The divergence is not laziness. Assembly genuinely produces
one of the three kinds of knowledge, and the feeling does not distinguish
them.
## 3. Why the confidence signal misfires
Effort is the proxy the mind uses for depth of understanding, and integration work is
genuinely effortful. Getting four components to agree on a data format, an auth scheme
and a deployment target is hard, absorbing, and produces a legitimate sense of
achievement. The proxy is not stupid. It is measuring the wrong quantity.
Two effects make it worse in current practice.
**3.1. The feedback loop is fast and one-sided.** A generated module that compiles and
passes the tests you asked for arrives in seconds, and the loop closes on "it works".
Nothing in that loop ever asks what happens when the upstream returns a 503 mid-stream.
**3.2. Review inherits the same bias.** A reviewer reads a diff for correctness against
the stated intent. If neither author nor reviewer has read the error paths, the review
confirms the happy path twice and records it as two independent confirmations.
Happy path, authoredread
Happy path, generatedskimmed
Error paths, authoredpartly
Error paths, generatedrarely
Figure 2. Ranked from code review, not measured. The bottom row is the claim; the others are context for it.
## 4. The remedy, which is not "write it yourself"
Refusing generated or borrowed code is not available and would not be correct. Every
engineer working today assembles from parts they did not write, and that has been true
since the first shared library. The remedy is to buy the third kind of knowledge
deliberately, because it is the only kind assembly never supplies for free.
**4.1. The prediction test.** Before merging, state in one sentence how this component
fails: what it does on timeout, on partial write, on malformed input, on a dependency
returning success with an empty body. If you cannot answer without running it, you have
assembled it and not read it. That is fine, and now you know which one you did.
**4.2. Read the error paths only.** A full read of generated code is often not worth the
time. A read of every branch that handles a non-success condition almost always is, and
it is a small fraction of the lines.
**4.3. Put the failure mode in the pull request.** One line, in the description. It makes
the gap visible to the reviewer, converts an assumption into a claim, and is the artefact
that pays off during the incident eighteen months later. See 5.11.
**4.4. Add one adversarial test per component, not per function.** Not coverage. One test
that does the rude thing: kills the connection halfway, sends the wrong content type,
returns 200 with an empty body.
## 5. The strongest objection
This has always been true and we have been fine. Nobody reads the
implementation of their TLS library, their JSON parser or their database driver, and the
industry works. If the argument does not explain why assembling from a well-tested
library is safe while assembling from generated code is not, it is nostalgia rather than
analysis. My attempted answer is that a widely used library has had its error paths
exercised by thousands of other people, and the resulting knowledge exists publicly in
issue trackers and postmortems even if I have not read it, whereas code generated for me
alone has a population of one and no such commons. I think that distinction is real, but
I cannot yet state where the population threshold sits, and without that the argument
does not give actionable advice about any specific dependency. That is the entire gap
between 0.75 and 0.9. A second objection I take seriously: the effect may be about
ownership rather than comprehension, in which case the remedy in Section 4 is aimed at
the wrong target.
## 6. What this paper does not claim
Generated code is not lower quality. In my experience it is often better than the median
hand-written equivalent on the happy path, and the engineers using these tools are not
less skilled than the ones who do not. Reading is not always worth the time either;
Section 4.2 exists precisely because it usually is not.
The claim is narrow. Assembly produces two of the three kinds of knowledge in Section 2,
the missing one is the one incidents require, and the feeling of understanding does not
report which of them you are holding.
--- 5.3 Algorithmic Homophily ---
url: https://mosthofaimran.com/papers/algorithmic-homophily/
state: revising
confidence: 0.6
revised: 2026-08-14
retires:
- A demonstration that technical opinion measured inside a subscription-shaped medium is no less varied than opinion measured outside it, on the same population and the same question, which would remove the effect this paper is about rather than its explanation.
- Evidence that deliberate exposure to disagreement, as described in Section 5, produces no measurable change in the accuracy of technical forecasts, which would leave the paper describing something real and useless.
- A mechanism that accounts for the private mailing list result in erratum 7.1 and predicts that ranking is nonetheless the dominant term, which would restore the original claim rather than the narrowed one.
Abstract. A ranked feed behaves like a cache with a hit-rate
objective and no invalidation policy. It returns what you already agreed with, faster
each time, and the sensation it produces is not comfort but consensus, which is why it is
hard to notice from inside. This paper is being revised. The mechanism proposed in
Section 3 was shown to be wrong in August 2026; the effect appears to be real and my
explanation of it was not. Confidence 0.60, down from 0.80. See erratum 7.1.
## 1. The claim, as it now stands
Two claims, which the first version of this paper did not separate, and separating them
is most of what the revision is for.
**The observation.** Practitioners inside a given medium converge in technical opinion
faster than the underlying evidence justifies, and the convergence is experienced as
consensus rather than as narrowing. I still hold this at roughly the confidence I started
with.
**The explanation.** That ranking algorithms optimising for engagement are the cause. This
is the part that broke, and it broke cleanly rather than partially, which is the good kind
of wrong.
## 2. The observation, and why it is hard to see
Convergence is invisible from inside because the mechanism that produces it also produces
its own corroboration. The same opinion arriving from eleven directions reads as eleven
independent confirmations. It is one confirmation, resampled.
what it feels like what it is
A B C D E A --+
| | | | | |
+--+--+--+--+ B --+-- one source,
| C --+ eleven arrivals
v D --+
consensus E --+
Figure 1. Independent confirmation and correlated resampling are
indistinguishable at the point of reading. Only provenance separates
them.
The engineering consequence is specific: technology choices acquire the appearance of
settled practice before the evidence exists, and the papers, benchmarks and postmortems
that would test them arrive two years later, by which point the choice is load-bearing.
Section 5.10 of this document is a retracted paper of mine that was produced exactly this
way.
## 3. The mechanism, under rewrite
This section is wrong and is being replaced. It is retained
rather than deleted, per Section 2.2 of the index draft. Two readers demonstrated in
August 2026 that the same convergence appears in closed, chronological mailing lists with
no ranking whatsoever, which the argument below cannot account for. Erratum 7.1 records
the correction and credits it.
The original argument ran as follows. Ranking optimises for engagement. Engagement
correlates with agreement, because disagreement costs attention and produces exit.
Therefore a ranked feed converges on your priors, and does so faster the more you use it.
The cache analogy did real work here: a cache with a hit-rate objective and no
invalidation policy will happily serve stale entries forever, because staleness is not on
the objective.
The mailing list result falsifies the necessity of ranking. Whatever produces the
convergence is present when ranking is absent.
**The candidate replacement**, which I am not yet confident enough to state as the
paper's position: the binding term is *selection*, not ranking. Choosing whom to read is
already a filter, it is applied once and revisited almost never, and it is applied on
criteria (clarity, seniority, agreeableness) that correlate with existing agreement.
Ranking then accelerates a process that subscription already started. If that is right,
the paper's advice barely changes and its target changes completely: the intervention
belongs at the subscription boundary rather than at the feed.
I do not yet have a way to distinguish the selection account from a third possibility,
which is that professional communities converge for ordinary social reasons that have
nothing to do with media at all. Until I can, this section stays marked.
## 4. What the revision does not change
The observation in Section 1 survives the correction, and so does the practical advice,
which is why the paper is being revised rather than retracted. Advice that survives the
falsification of its own mechanism should be treated with suspicion, and I am treating it
with suspicion: it may be surviving because it stands on its own, or because it was
never load-bearing on the mechanism in the first place.
## 5. The practice, offered at 0.60
- **Write down what would change your mind before reading.** This is the same discipline
the retirement conditions on this site enforce, applied to consumption rather than to
publication. It is the only intervention in this list I am confident about, because its
value does not depend on which mechanism is correct.
- **Audit the subscription boundary, not the feed.** If the selection account is right,
this is where the whole effect enters. Once a quarter, list who you read and ask which
of them has told you something you did not want to hear.
- **Prefer artefacts with provenance.** A postmortem, a benchmark with a methodology, a
paper with a retirement condition. These are harder to resample, because the source is
attached.
- **Count arrivals, not sources.** When a position reaches you from many directions, the
useful question is how many distinct pieces of evidence sit behind it, which is usually
one.
## 6. The strongest objection
This may be an ordinary property of professional communities,
described in a technological vocabulary that adds nothing. Engineers converged on
opinions before ranked feeds, before mailing lists, and before the internet, through
conferences, employers and textbooks. If the effect appears with ranking, without ranking,
and plausibly without any medium at all, then "algorithmic" in the title is doing no work
and the honest paper is a much older one about professional consensus. I cannot currently
rule this out. It is the reason the confidence is 0.60 rather than 0.70, and it is the
reason Section 3 is marked rather than quietly rewritten.
## 7. What this paper does not claim
It does not claim consensus is wrong; most consensus is correct and cheaply acquired. It
does not claim engagement optimisation is malicious. It does not, as of the August 2026
revision, claim that ranking causes the effect, and any quotation of the earlier version
saying so should be treated as withdrawn. And it does not claim a fixed timeline for the
rewrite, because the replacement mechanism is not yet good enough to publish.
--- 5.4 The Easy Button Tax ---
url: https://mosthofaimran.com/papers/easy-button-tax/
state: holding
confidence: 0.85
revised: 2026-08-14
retires:
- A widely adopted abstraction that removed a class of friction and whose failure modes are demonstrably cheaper to diagnose than the friction it replaced, measured in operator time during incidents rather than in developer time during authoring.
- Evidence that the cost transfer described in Section 2 does not occur where the author and the operator are the same team, which would reduce this to an argument about organisational structure rather than about abstraction.
- A convenience layer shipping with an escape hatch that is exercised in its own test suite as a first-class path, adopted at scale, showing that the tax is a choice rather than a property.
Abstract. Friction is information. An abstraction that removes a
step also removes the moment where the constraint behind that step was learned. The
constraint does not disappear; it reappears at incident time, in front of a person who
did not choose the abstraction, priced in hours they do not have. This is not an argument
against abstraction. It is an argument for reading the invoice before signing, and for
insisting that every easy button ship with a tested escape hatch.
Confidence 0.85. The missing 0.15 is Section 5, where the argument still cannot
tell a good abstraction from a costly one in advance.
## 1. The claim
Every convenience layer makes a trade with a specific shape: it converts a large number
of small, predictable, design-time costs into a small number of large, unpredictable,
incident-time costs. The trade is often correct. It is almost never priced, because the
two sides of it are paid in different currencies by different people at different times.
The word "tax" is chosen carefully. A tax is not a scam. It is a known charge on a
transaction, and the failure here is not that the charge exists but that it is
undisclosed at the point of sale.
## 2. The three transfers
Transfer
From
To
In time
Design time, when the system is calm, the decision is reversible and the person has context
Incident time, when the system is degraded, the decision is urgent and the context has to be rebuilt from logs
In person
The author, who chose the abstraction and understands what it hides
The operator, who inherited it, and who is often on a different team, in a different timezone, three years later
In kind
Many small comprehension costs, each cheap and each teaching something
One large diagnostic cost, expensive and teaching nothing except the shape of this particular abstraction's internals
The third transfer is the one that compounds. Learning a constraint by hitting it during
authoring produces knowledge that transfers to the next system. Learning it by reading a
stack trace through four layers of framework at 03:00 produces knowledge about that
framework's internals, which is worth very little the moment the framework is replaced.
cost
^
| ..... friction paid at design time
| . . . . . . . . .
| --------------------------------- calm
|
| ##
| ## incident
| ................................##
+------------------------------------> time
abstraction adopted first
real failure
Figure 1. The same total cost, differently distributed. The right-hand
column is paid in a currency the left-hand column was not.
## 3. Diagnosing an easy button
The useful question is not "is this abstraction good" but "what did it decide on my
behalf, and can I see the decision". Three tests, in increasing order of how much they
tell you.
**3.1. The naming test.** Can you name the thing it hides? If the answer is a vague
category ("it handles the networking") rather than a specific mechanism ("it retries idempotent
requests three times with deterministic backoff and no budget"), you do not know what you
bought. See 5.6 for what that particular blank cheque costs.
**3.2. The escape hatch test.** Is there a documented way to drop below the abstraction
for one call, and is that path exercised in the library's own tests? An escape hatch that
exists in the documentation and not in the test suite is a plan, not a mechanism. This is
the single strongest predictor I have found of whether an abstraction will be survivable
in year three.
**3.3. The incident test.** Read one public postmortem from a team that hit this
abstraction's failure mode. If none exists, either the abstraction is new or its failures
are being resolved by vendor support tickets, which means the diagnostic knowledge is not
in the commons and you will be rebuilding it yourself.
Names what it hidescheap
Escape hatch, testeddecisive
Public failure recordrare
Figure 2. The bars rank how much each test tells you against how often
it can be satisfied. Ranked, not measured.
## 4. The outcome this argues for
Not "avoid convenience". The outcome is a disclosure practice, and it is small enough to
adopt this week.
- **Write the transfer down at adoption time.** One line in the design document: what
this removes, what it hides, and who pays when it fails. It takes ten minutes and it is
the artefact the operator will want in three years.
- **Require the escape hatch before adoption, not after.** If dropping one call below the
abstraction requires forking the library, the abstraction is not a layer, it is a
ceiling.
- **Exercise the hatch once, in CI.** A single test that goes around the convenience path
keeps it alive. Escape hatches rot silently otherwise.
- **Treat the first incident as the invoice arriving.** Record the price in the
postmortem so the next team can compare it against the convenience it bought. See 5.11.
## 5. The strongest objection
This is a general argument against progress, and progress has
mostly been right. Garbage collection, optimising compilers, managed relational
databases and TLS libraries are all easy buttons, all hide enormous complexity, all
relocate friction to incident time, and all were correct. If the argument cannot
distinguish those from a badly designed convenience wrapper, it distinguishes nothing.
My attempted distinction is that the good cases hide a mechanism that is genuinely
universal and genuinely solved, so the hidden constraint is nearly never the thing that
fails, while the expensive cases hide a mechanism that is domain-specific and still
contested. I cannot yet state that cleanly enough to apply it in advance rather than in
hindsight, and applying it in hindsight is worth very little. That gap is the whole 0.15,
and it is why this paper is not at 0.95.
## 6. What this paper does not claim
The tax is usually worth paying, and nothing here argues for shallow abstractions or for
building it yourself. The authors of convenience layers are not the target either. In the
cases that cost the most, the author was unusually careful, which is exactly why the
abstraction was adopted widely enough to cost anything. The claim is about disclosure,
and about who receives the bill.
--- 5.5 On-Premise Is Not a Downgrade ---
url: https://mosthofaimran.com/papers/on-prem/
state: holding
confidence: 0.9
revised: 2026-08-14
retires:
- A team that adopted the single-artifact discipline at design time and can show, over two years, that it consumed more total engineering hours than maintaining separate cloud and on-premise builds.
- Regulated estates routinely permitting outbound connections to vendor control planes, which would make the assumption list in Section 2 historical rather than current.
- A cloud-only system of comparable complexity demonstrating equivalent dependency hygiene, reproducibility and upgrade safety without any sovereignty constraint forcing it.
Abstract. On-premise delivery is expensive when it is treated as a
cloud deployment with things taken away. Treated as a constraint adopted at design time
it is not a downgrade, and the discipline it forces (one artifact, explicit
dependencies, an offline supply chain, configuration as data) improves the cloud build
as well. The cost of sovereignty is not hardware. It is the number of assumptions you
can no longer make, and that number is finite, enumerable and mostly known on day one.
Confidence 0.90. Section 5 holds it there: elastic workloads are a real
exception and I have not sized how large that exception is.
## 1. The claim
The industry describes on-premise work in the vocabulary of loss. Legacy. Regression.
Enterprise tax. That vocabulary is not neutral, and it produces a specific engineering
failure: teams build for the cloud, ship, and then attempt to subtract their way to an
on-premise release. Subtraction is where the cost is. Almost every hour I have watched
burned on sovereign delivery was spent removing an assumption that had been free to
avoid at the start and expensive to remove later.
The claim is narrow and it is about ordering. Sovereignty adopted as a constraint before
the first architectural decision costs roughly what any other constraint costs.
Sovereignty adopted as a port costs several multiples of that, and the multiple grows
with the age of the codebase.
## 2. The assumption ledger
What actually changes between a public cloud estate and an air-gapped one is a list. It
is shorter than the folklore suggests, and each entry has a known design response.
Assumption removed
Design response
Outbound network egress
No vendor control plane, no licence phone-home, no telemetry upload, no package fetch at deploy time. Everything the system needs at run time is inside the bundle.
Managed services
Every managed dependency sits behind an interface the system owns, with at least one self-hosted implementation that is exercised in CI rather than kept as a claim.
Vendor-driven upgrades
Upgrades become signed bundles applied by the operator, with a rollback that is tested rather than documented.
Observability as a service
Metrics, logs and traces terminate inside the estate. Support debugging happens on evidence the customer exports deliberately, not on a dashboard you can open.
Elastic capacity
Capacity becomes a stated failure point rather than an autoscaling policy. See Principle 4.7.
You, as the operator
The operator is a person you will never meet, working a maintenance window at 02:00, holding a printed runbook. This is the entry that changes the most and gets the least attention.
That last row is the one that decides whether a sovereign product is viable. Every other
entry is an engineering problem. The operator is a design problem, and the failure mode
is a system that is correct and unoperable.
## 3. Why the constraint pays back
The interesting result is not that these responses are achievable. It is that each of
them is independently good practice, and a team that adopts them under sovereign
pressure ends up with a cloud build that is measurably better than the one they would
have shipped without it.
source -> reproducible build -> signed bundle
|
+--------------------+
| |
public cloud air-gapped site
(same bytes) (same bytes)
| |
config as data config as data
Figure 1. One artifact, two destinations. The moment the two paths
diverge, the second one starts rotting, because only the first is exercised
daily.
Three specific payoffs, in the order I have seen them arrive.
**3.1. Dependency honesty.** A build that must run with no egress cannot pretend about
what it depends on. Transitive fetches, implicit base images and "it works on the runner"
all fail immediately rather than in year three.
**3.2. Reproducibility becomes non-optional.** Shipping a bundle to a site you cannot
reach means the bundle has to be the whole truth. That forces reproducible builds, which
in turn makes cloud incidents diagnosable, because the artifact in production is
byte-identical to one you can rebuild.
**3.3. Configuration stops living in the environment.** When the environment cannot be
inspected, configuration has to be data that travels with the deployment and can be
diffed. This removes a large class of cloud incidents whose root cause is a variable set
by hand in a console two years ago.
Design-time constraintbaseline
Port at v1higher
Port at v3highest
Figure 2. The shape of the cost, not its magnitude. I do not have
defensible figures for the ratios and will not invent them; see
docs/PLACEHOLDERS.md.
## 4. What this looks like as a rule set
- **One artifact.** If cloud and on-premise builds differ, they differ in configuration
data, never in code paths selected at build time.
- **Every managed dependency behind an owned interface**, with a self-hosted
implementation running in CI on every commit.
- **The supply chain is offline-first.** Vendored, hashed, and verifiable without a
network. A build that needs the internet is a build that cannot ship to a bank.
- **Upgrades are signed bundles with a tested rollback**, applied by a stranger, in one
maintenance window, with no interactive prompts.
- **Capacity is stated as the point at which the system fails**, not as a target it
meets. The operator needs to know where the edge is, because they cannot add nodes.
- **The runbook is a deliverable**, versioned with the code, and it is wrong until
somebody who did not write the system has followed it end to end.
## 5. The strongest objection
For genuinely elastic workloads, on-premise is a real downgrade and
calling it a constraint is a euphemism. This is correct and it bounds the paper. A
workload whose value comes from absorbing a hundred-fold burst for four hours a year is
worse on fixed hardware, and no amount of design discipline recovers that. The paper
holds for systems with predictable load envelopes, which in my experience is most
regulated workloads, but "most" is doing work in that sentence and I have not
quantified it. There is a second cost I have understated: the single-artifact discipline
slows the first six months, and for a team still searching for product fit that slowdown
can be fatal. The constraint pays back over years. Not every project has years.
## 6. What this paper does not claim
On-premise is not cheaper. Total cost of ownership is usually higher, and the customer is
usually paying it deliberately, for reasons of jurisdiction, audit or counterparty risk
that have nothing to do with engineering. Nothing here says cloud teams are
undisciplined, only that the sovereign constraint removes the option of skipping the
discipline, which is a different and much weaker statement about them.
The assumption ledger in Section 2 is not complete. It is the list I have needed so far,
across six sites, and every new estate has added to it.
--- 5.6 The Retry Storm You Built On Purpose ---
url: https://mosthofaimran.com/papers/retry-storm/
state: holding
confidence: 0.95
revised: 2026-08-14
retires:
- A fleet of a thousand or more clients running deterministic exponential backoff with no jitter and no retry budget, surviving a sixty second dependency outage with no correlated arrival spike, measured at the dependency rather than at the client.
- Evidence that the default settings of the major client libraries now ship full jitter and a caller-side budget, which would make this a paper about a solved problem rather than a live one.
- A queueing analysis showing that at realistic client counts the benefit of jitter is dominated by other recovery effects, so that removing it changes nothing measurable.
Abstract. Exponential backoff is presented as a politeness
mechanism. Without jitter it is a synchronisation mechanism. A single shared fault starts
every client's timer at the same instant, deterministic delays preserve that alignment
through every subsequent round, and the recovery attempt arrives as a series of spikes
that grow with the length of the outage. The failure is not that engineers do not know
about jitter. It is that retries are budgeted as a count per call instead of as a
fraction of forward traffic, and that nobody writes down which layer is allowed to retry.
Confidence 0.95. The missing 0.05 is Section 5: the sharper version of this
paper is about budgets alone, and I have not rewritten it that way yet.
## 1. The claim
A retry is a load-generating decision made by a component that has just been told the
system is under stress. That is the whole problem in one sentence. Every other property
of retry behaviour follows from it.
Exponential backoff is the standard mitigation and it is a good one. It is also
incomplete in a specific, mechanical way. Backoff controls *when* one client retries. It
says nothing about whether a thousand clients retry at the same moment. If the delay
schedule is deterministic, and the fault that triggered it was shared, then the schedule
does not spread the load. It preserves the alignment the fault created and carries it
forward, round after round, for as long as the outage lasts.
## 2. The mechanism
A dependency returns errors starting at `t=0`. Every in-flight caller observes the
failure within one round-trip time of each other, which on a healthy internal network is
a window of a few milliseconds. That is the synchronising event. From then on, a
deterministic schedule of 1s, 2s, 4s, 8s keeps the whole population inside that same few
milliseconds at every retry boundary.
0s 1s 2s 4s 8s
| | | | |
no jitter |#######|#######|#######|#######|
fault R1 R2 R3 R4
^ ^ ^ ^
every caller arrives in the same window
full jitter |#.#..#.|..#.#..|#..#..#|.#..#.#|
arrivals spread across the whole gap
Figure 1. Deterministic backoff preserves the alignment the fault
created. Jitter destroys it. Both schedules have the same mean delay.
The dependency therefore recovers into a square wave rather than into a ramp. Its first
moment of health is also the moment of peak concurrent arrival, so it fails again, which
re-synchronises the population, which produces the next spike. The system has found a
stable oscillation and will stay in it until something outside the loop intervenes.
Two amplifiers make this worse than the single-layer picture suggests.
**Amplification through layers.** Retries compose multiplicatively. Three attempts in the
SDK, inside three attempts at the gateway, inside three attempts in the calling service,
is not nine requests and it is not three. It is twenty-seven, and no single layer's
configuration looks unreasonable on its own.
1 retrying layer3×
2 retrying layers9×
3 retrying layers27×
Figure 2. Worst-case request amplification for three attempts per layer.
The multiplier is the product of the layers, not the sum.
**The load grows while the outage lasts.** Callers that would have arrived during the
outage do not disappear. They queue in the client, in the connection pool, in the
upstream's own inbound buffer. The longer the dependency is down, the larger the
population that arrives in the first post-recovery window.
## 3. What it looks like in production
The signature is specific enough to be diagnosed from a graph without reading any code.
Signal
What a retry storm looks like
Arrival rate at the dependency
Periodic spikes at 1s, 2s, 4s, 8s offsets from the fault, not a smooth ramp.
Ratio of attempts to distinct calls
Rises well above 1.0 and stays there. This is the single most useful number and almost nobody emits it.
Recovery shape
Health flaps. The dependency comes up, dies inside one window, comes up again.
Client-side latency
p99 grows by the sum of the backoff schedule, so it looks like the dependency got slow when in fact the caller is waiting on its own timers.
The third row is why this is rarely caught in testing. A load test drives a synthetic
client population with independently random start times, which is jitter arriving by
accident. The synchronising event is absent from the test because the test never has a
single shared fault.
## 4. The remedy, in the order it matters
The order is deliberate. The first item is the cheapest and the third is the one that
actually holds.
**4.1. Jitter, and specifically full jitter.** Sleep a uniform random value in
`[0, base * 2^n]` rather than the value itself. Equal jitter and decorrelated jitter are
both defensible; the important property is that the delay is drawn from a distribution
rather than computed, so that no two callers share a wake time except by coincidence.
**4.2. A retry budget at the caller, expressed as a fraction.** Not "three attempts per
call". A token bucket refilled from forward traffic, permitting retries only while
retries stay under roughly ten percent of successful requests over a sliding window. A
count-based limit rises with load exactly when it should fall. A fraction-based budget
falls with success exactly when it should.
**4.3. Exactly one retrying layer, named in the design document.** This is the rule that
survives reorganisation, because it is a written decision rather than a configuration
value. Every other layer converts failures into errors and returns them. If nobody can
say which layer owns retries, the answer is all of them.
caller -> gateway -> service -> store
[R] [ ] [ ] [ ]
|
+-- retries here, with a budget and jitter
every other hop fails fast and reports
Figure 3. One retrying layer. The choice of which layer matters less
than the fact that it is written down.
**4.4. Emit the attempt-to-call ratio.** If the number is not on a dashboard, the storm
is invisible until it is an incident. See 5.13.
## 5. The strongest objection
Jitter is textbook, and this paper is scolding people for something
they already know. That is close to right, and it is what keeps the confidence off
1.0. Nearly every engineer who reads this can define jitter. The claim survives on a
narrower footing: knowing about jitter has not translated into budgets, and budgets are
the part that actually bounds the blast radius. Jitter spreads the same total load;
only a budget reduces it. I hold 0.95 rather than 1.0 because the sharper version of
this paper would be about budgets alone, and I have not yet rewritten it that way. There
is also a real cost I have understated: full jitter adds tail latency to the common case
of a single transient error, and for small fleets that cost can exceed the benefit.
## 6. What this paper does not claim
Retries are not the problem. A system without them converts every transient fault into a
user-visible error, which is worse than a storm you can bound. Jitter alone is not the
remedy either, and Section 4.2 exists because of that. The ten percent figure is a
starting point that has held for me across several systems rather than a derived
constant, and I would expect it to be wrong for anything with a very different ratio of
read to write traffic.
Nor is any of this novel. The mechanism is well described in the literature. What the
paper claims is a gap between that description and what is actually configured in
production, and erratum 7.3 records me falling into that gap myself, in code I reviewed
and approved.
--- 5.7 Chesterton's Fence Has a Git Blame ---
url: https://mosthofaimran.com/papers/chestertons-fence/
state: holding
confidence: 0.8
revised: 2026-08-14
retires:
- A codebase of substantial age where the instrument-then-remove procedure in Section 4 produced no measurable reduction in regressions from cleanup work, compared against a matched period of direct deletion.
- Evidence that guard clauses whose recorded reason has decayed are, in aggregate, no more likely to be load-bearing than newly written ones, which would make the caution in this paper an expensive superstition.
- Tooling that reliably reconstructs the intent behind a change from the surrounding artefacts, at accuracy high enough that the decay ladder in Section 2 stops mattering.
Abstract. The parable says do not remove a fence until you know
why it was built. In software the answer is usually recoverable, for a while, from the
commit, the pull request, the issue and the incident it followed. The parable is
therefore easier to obey here than anywhere else, and it is still routinely disobeyed,
because the recoverable context decays on a schedule nobody plans around. This paper is
about the case the parable does not cover: when the history is one line and a date, and
the honest answer to "why is this here" is that nobody knows.
Confidence 0.80. Section 5 is why it is not higher: I cannot separate a
load-bearing fence from a decorative one before instrumenting it.
## 1. The claim
Two claims, and the second is the one that is actually contested.
The weak claim: before deleting a guard clause, a retry, a sleep, a null check or a
special case, spend ten minutes on `git log -S` and the linked pull request. This is
cheap, it works more often than people expect, and it is not interesting.
The strong claim: the context that would answer the question decays predictably, and the
decay is fast enough that the parable's advice is unavailable for most code older than
about three years. What is needed for that case is a procedure, not more diligence.
## 2. The decay ladder
Every reason for a line of code lives somewhere. The places are ordered by how long they
survive, and the ordering is stable across every organisation I have worked in.
survives longest
| inline comment naming the failure
| test asserting the behaviour
| commit message with the reason
| pull request description
| issue tracker entry
| incident channel / chat thread
| the person who wrote it
survives shortest
Figure 1. The decay ladder. Everything below the third rung depends on
a system, an account or a person outliving the code, and none of them
reliably do.
The bottom four rungs share a property: they are outside the repository. They depend on a
vendor account still existing, a chat retention policy, a tracker migration that preserved
comments, or a person still answering email. The repository is the only artefact with the
same lifetime as the code, which is why the top three rungs are the only ones worth
relying on.
Inline commentcode’s life
Commit messagerepo’s life
Pull request bodyvendor’s life
Issue trackermigration
Chat threadretention
The authortenure
Figure 2. The bars rank expected survival rather than measured
half-lives. The ranking is the claim; the lengths are only a drawing of it.
## 3. The obligation this puts on the author
Most writing about Chesterton's fence addresses the person removing it. The larger gain
is on the other side, because it is cheap at the moment of writing and impossible
afterwards.
A guard clause should carry its reason at the top of the ladder, in one line, naming the
failure rather than the behaviour:
// Rejects zero-length batches. The 2024-11
// ingest incident: an empty batch advanced the
// offset without a write, so replay skipped the
// window silently.
// Test: batch_empty_does_not_advance_offset.
if (batch.isEmpty()) return
Figure 3. Four lines that make the fence removable by someone who was
not there. The test name is the load-bearing part.
A test that fails when the guard is removed is better than any comment, because it
enforces rather than explains. The comment is for the case where the reason is not
expressible as an assertion, which is more common than test-first advice admits: rate
limits, ordering assumptions about an external system, and workarounds for defects in
software you do not control.
## 4. The procedure for the decayed case
When the history is genuinely thin, the parable gives no guidance beyond "do not remove
it", which taken literally means codebases only accumulate. The alternative is to convert
the unknown into an observation.
§
Step
1
Instrument, do not delete. Leave the guard in place and emit a counter with a distinguishing label every time it fires. This is a small, reversible, obviously safe change.
2
Wait one full business cycle. Not a week. Whatever period contains your month-end close, your quarterly batch, your annual reconciliation. Fences are usually built for the rare path, which is exactly the path a two-week observation misses.
3
Read the fires. If the counter is non-zero, you now have the reason, expressed as the inputs that reach it, which is better evidence than the original commit message would have been.
4
If it is zero, remove with the evidence attached. The commit message says the counter ran for the named period at the named volume and never fired. That message is now the top rung of the ladder for whoever revisits this.
5
Keep the rollback cheap for one more cycle. Removal is a change like any other, and it deserves the same rollback plan as a feature.
The procedure has a cost, and the cost is the objection.
## 5. The strongest objection
This turns every cleanup into a quarter-long project, and most fences
are cargo cult. Both are substantially true. A large fraction of guard clauses in any
old codebase are defensive habit, copied from a neighbouring function, protecting against
nothing. Applying Section 4 to all of them would make deletion so expensive that nobody
deletes, and a codebase that cannot shrink is its own failure mode. I do not have a
reliable test that separates a load-bearing fence from a decorative one before the
instrumentation runs, which is the entire reason this sits at 0.80. The partial answer I
use is to apply the procedure only where the guard touches money, ordering, retention or
an external contract, and to delete freely elsewhere. That heuristic is a judgement call
wearing a rule's clothing, and I know it.
## 6. What this paper does not claim
Old code has no special claim on survival, and `git blame` is not sufficient: Section 2
exists because it usually is not. The ordering in Figure 1 is not a measurement. It is
what I have seen hold everywhere I have looked, which is a weaker thing.
The procedure in Section 4 is not novel either. It is ordinary feature-flag practice
pointed at a deletion instead of a release, which is the one place almost nobody points
it.
--- 5.8 RAG Is a Search Problem in a Trench Coat ---
url: https://mosthofaimran.com/papers/rag-search/
state: holding
confidence: 0.7
revised: 2026-08-14
retires:
- A published evaluation on a realistic corpus in which swapping the embedding model, holding chunking, query construction and retrieval strategy fixed, produces a larger gain in answer accuracy than fixing chunking while holding the model fixed.
- Context windows and attention costs reaching a point where whole-corpus prompting is economically routine, which would remove the retrieval stage this paper is about rather than improve it.
- Evidence that retrieval recall of the answer-bearing passage is not the binding constraint in production systems, for example generation reliably recovering answers absent from the retrieved context.
Abstract. Retrieval-augmented generation is an information
retrieval pipeline with a language model at the end of it. Most of the quality is decided
by the classic parts: how documents are split, what metadata survives the split, how the
query is constructed, and whether ranking is hybrid. Those decisions are usually made in
one afternoon by whoever set the system up, and then never revisited, while the team
spends the following two quarters comparing embedding models. This paper is deliberately
narrower than the claim I got wrong in 5.10, and erratum 7.2 is the reason.
Confidence 0.70. Section 5 is why it is not higher: long context may retire the
retrieval stage rather than improve it.
## 1. The claim
If the passage containing the answer is not in the retrieved set, nothing downstream
recovers it. Not a better reranker, not a larger model, not a more elaborate prompt. That
is a hard ceiling, and it is set by decisions that happen before any embedding is
computed.
The claim is therefore about attention, not about technology: the marginal engineering
hour in most retrieval systems is better spent on chunking, query construction and
evaluation than on the embedding model, and teams reliably spend it the other way because
the model is the part with a leaderboard.
## 2. Where the losses actually happen
document
| split <- most loss enters here
v
chunk (metadata kept? structure respected?)
| embed <- the part everyone tunes
v
vector -> index <- recall ceiling set here
|
query (as typed? expanded? filtered?)
| retrieve k <- k set once, never tuned
v
rank (dense only? hybrid?)
|
v
assemble context <- order, truncation
|
v
generation
Figure 1. Six decisions before generation. Five of them are ordinary
information retrieval and predate the current vocabulary by decades.
**2.1. Splitting.** Fixed-size windows with a fixed overlap are the default in every
starter template, and they cut through tables, headings, list items and the sentence that
defines the term used in the next paragraph. A split that respects document structure
(sections, table boundaries, list integrity) usually beats any model change, and it costs
a day.
**2.2. Metadata loss.** The chunk arrives at the index having forgotten which document,
which version, which section, which effective date and which tenant it came from. Every
one of those is a filter that would have removed most of the false positives, and the
loss is silent because the pipeline still returns plausible results.
**2.3. The query is used raw.** Users type fragments, misspellings and internal jargon.
The corpus is written in formal prose. Dense retrieval is good at bridging that gap and
lexical retrieval is good at exact identifiers, part numbers and error codes, which is
precisely where dense retrieval fails and precisely what users paste in.
**2.4. `k` was chosen once.** Almost always 3, 5 or 10, on the first day, and it is never
tuned against a measurement because there is no measurement.
## 3. The measurement that changes the conversation
Retrieval quality and answer quality are different numbers and must be measured
separately. Almost every team measures the second and infers the first, which makes every
regression ambiguous.
§
Step
1
Build an evaluation set of real questions, fifty is enough to start, taken from what users actually asked rather than what the corpus makes easy.
2
Label the answer location, not the answer. For each question, record which passage of which document contains it. This is the expensive part and it is done once.
3
Measure recall@k of the answer-bearing passage. This number is the ceiling on the whole system. If it is 0.6, no amount of generation work takes the system above 0.6.
4
Only then measure answer quality, and treat any gap between recall and answer accuracy as a generation problem rather than a retrieval one.
Once step 3 exists, the argument about embedding models resolves itself empirically in an
afternoon, in either direction, which is the outcome this paper actually wants.
Structure-aware splittinglarge
Metadata filterslarge
Hybrid lexical + densesolid
Tuning k, rerankingmoderate
Newer embedding modelsmall
Figure 2. My ordering of expected marginal gain, not a measurement.
It is a hypothesis this paper asks you to test with Section 3, and it is exactly
the kind of ordinal claim that ought to carry a confidence value.
## 4. The order of work this argues for
1. Evaluation set with labelled answer locations. Nothing else is decidable without it.
2. Fix splitting so it respects document structure, and keep document, section, version
and tenant on every chunk.
3. Add lexical retrieval alongside dense, and fuse the rankings. Identifiers and error
codes stop disappearing.
4. Use the metadata as filters before ranking rather than as display fields after it.
5. Tune `k` and add reranking against the measurement from step 1.
6. Then, and only with a number to compare against, consider the embedding model.
## 5. The strongest objection
Long context may make the retrieval stage vestigial. If it becomes
economical to put a whole corpus, or a whole document set, in front of the model, then
splitting strategy stops being a quality decision and becomes a cost decision, and this
paper is about a transitional period rather than about a property of the problem. I do
not know how to weigh that, and it is most of the reason this sits at 0.70 rather than
higher. There is a second objection I take seriously: my ordering in Figure 2 comes from
systems with structured, versioned, tenant-scoped corpora, where metadata is unusually
valuable. On a flat corpus of undifferentiated prose the metadata rows collapse and the
ordering may invert. And I am aware that the last time I made a confident claim in this
area I had to retract it in full, which is recorded in erratum 7.2 and is the reason this
paper is scoped to a measurement practice rather than to a prediction about
technology.
## 6. What this paper does not claim
Embedding models matter, and vector indexes are necessary. Saying otherwise about the
second of those is the over-claim that retired 5.10, and I am not making it again.
Figure 2 does not generalise; it is labelled a hypothesis because that is what it is.
Classical information retrieval does not solve this on its own either. The claim is only
that the classical parts are where the unspent engineering hours are, and that a team
without the measurement in Section 3 has no way to find out whether that is true of their
corpus.
--- 5.9 The Ship of Theseus Passes Its Integration Tests ---
url: https://mosthofaimran.com/papers/ship-of-theseus/
state: draft
confidence: 0.55
revised: 2026-08-14
retires:
- A completed incremental migration of substantial size where no identity declaration was made, and where ownership, invariants and the decommissioning of the old path nonetheless resolved cleanly within a year of the last route moving.
- Evidence that end-to-end invariants are preserved by route-level verification in practice, which would remove the specific decay this paper is worried about.
- A demonstration that the residual old system is retired at similar rates whether or not a decommissioning date was declared in advance, which would make Section 4 ceremony.
Abstract. Incremental migration removes the risky cutover, which
is its entire justification and a real one. It also removes the moment at which anyone
verifies the system as a whole, declares the old guarantees ended, and names who owns the
new ones. This paper argues that the missing moment has costs, and that they are paid
quietly: invariants that were only ever true end to end, ownership that never transfers,
and an old path that is never decommissioned because no date was ever set for it.
This is a draft. The mechanism is stated, the evidence is thin, and the objection
in Section 5 is unanswered. Confidence 0.55.
## 1. The claim
A strangler-fig migration proceeds route by route. Each move is small, reversible, and
independently verified, which is why the pattern works and why it has largely replaced
the big-bang rewrite. Nothing in this paper disputes that.
What the pattern does not produce is a point in time at which someone says: the system is
now the new system, its guarantees are these, and this person owns them. In a big-bang
cutover that moment is unavoidable and expensive, and its expense is what buys the
verification. Incremental migration makes the moment optional, and optional
organisational moments do not happen.
## 2. What decays in the gap
**2.1. Invariants that were only ever end-to-end.** Consider a property like "every
accepted order appears in exactly one settlement batch". In the old system that held
because one process owned both sides. Route-level verification checks that the new order
path matches the old order path and that the new settlement path matches the old
settlement path. It does not check the property that spans them, because that property
was never a route.
old system invariant held here
[ orders -- settle ] <---- one owner, one process
during migration
[ orders ] --> new each route verified
[ settle ] --> old against its own old half
^
+-- nobody verifies the span
after
[ orders -- settle ] invariant assumed, not checked
Figure 1. Route-level equivalence does not compose into system-level
equivalence, and the gap is invisible while both halves pass.
**2.2. Ownership that never transfers.** The old system has an owner. The new one has a
migration team. When the last route moves, the migration team disbands and ownership
arrives at whoever is nearest, usually by accident, usually discovered during the first
incident.
**2.3. The old path that never dies.** A residual route left for a "long tail" client, a
batch job, a reconciliation script. It has no owner and no decommissioning date. It
accumulates the property of being the thing nobody understands, which is where 5.7 picks
up.
**2.4. The documentation describes neither system.** During migration every document is
provisional. Provisional documents are not maintained, and the migration is long enough
that the habit of not maintaining them outlives it.
Route-level correctnessverified
Spanning invariantsassumed
Ownershipimplicit
Old path retiredpending
Figure 2. What incremental migration verifies well against what it
leaves open. An impression rather than a finding; this is a draft, and the bars
are part of what Section 6 is asking about.
## 3. Why "it is done when the last route moves" is not enough
Because the last route is chosen by difficulty, not by importance. Migrations move the
easy traffic first, which means the final routes are the ones with the most unusual
requirements and the least understood behaviour, and the project reaches its lowest
morale and highest cost at exactly the point where the remaining work is hardest to
verify. "Ninety percent migrated" is a statement about routes and almost never a
statement about risk.
## 4. The proposal: identity by declaration
The remedy I am proposing, and the part I am least sure of, is to reintroduce the moment
deliberately, without reintroducing the risky cutover that the pattern exists to avoid.
It is a document and a date, not a deployment.
§
Declaration
1
Name the spanning invariants at the start, before the first route moves, and build a check for each that runs against the live system throughout the migration rather than against either half.
2
Publish a route ledger: every route, its state, its verification, its owner. One page. It is the only honest answer to "how far along are we".
3
Declare a date on which the old guarantees end, and name the person to whom the new ones transfer. This is the moment. It costs a meeting.
4
Set the decommissioning date before the migration starts, with an owner, and treat slipping it as a decision that needs a reason rather than as a default.
## 5. The strongest objection, unanswered
The missing moment may not matter, and wanting one may be aesthetic
rather than operational. This is the objection that keeps the paper at 0.55 and in
draft. Gradualism's entire benefit is that there is no discontinuity, and asking for a
declaration could be nostalgia for the ceremony of a cutover dressed up as a risk
argument. Every cost I list in Section 2 has an alternative explanation that has nothing
to do with the missing moment: spanning invariants decay in systems that never migrate at
all, ownership drifts under reorganisation regardless (see 5.12), and old code survives
for ordinary reasons of priority. Nothing here separates the migration-specific effect
from that background rate, which leaves a plausible mechanism standing on no evidence. That is what a draft is, and it is why the confidence is where
it is.
## 6. Open questions
Stated plainly, because this document is not finished and pretending otherwise would be
the failure mode the rest of the site is about.
- Is the spanning-invariant decay measurable, and does it differ from the background rate
in comparable systems that did not migrate?
- Does the declaration in Section 4 change behaviour, or does it become a ceremony that
is performed and ignored?
- Is there a version of the route ledger that survives the migration and becomes the
ownership document, or does it die with the project?
- Does this apply below some size? A four-route migration probably needs none of this,
and I do not know where the threshold is.
## 7. What this paper does not claim
Big-bang rewrites are not better. They are worse, and the pattern this paper criticises
is the correct default. Incremental migrations do not fail; most of the ones I have seen
succeeded on their own terms.
The claim is only about what they leave behind, and at 0.55 it is a claim I would not
want quoted without its confidence value attached to it.
--- 5.10 Vector Databases Are a Fad ---
url: https://mosthofaimran.com/papers/vector-db-fad/
state: retracted
confidence: n/a
revised: 2026-08-14
retires: NONE STATED. This entry is not an argument.
Retracted in full on 2025-11-14. The text below is the original
argument, retained rather than deleted, per Section 2.2 of the index draft. It is wrong.
Section 5 of this page records what failed and what survived. Do not quote any part of
Sections 1 to 4 without this notice attached.
## 1. The claim, as originally published
Vector search is a feature, not a product. The index structures involved are published
algorithms with open implementations, the storage layer is a solved problem, and the query
patterns are narrow. General-purpose databases will absorb the capability within two
years, at which point a separate stateful system exists only to serve a workload its
neighbour can already handle. Standing up a second database with its own operational
model, backup story, upgrade cadence and on-call knowledge is a cost that this workload
does not justify.
## 2. The supporting argument, as originally published
Three points were offered. First, that approximate nearest neighbour search is a
well-understood problem with published algorithms, so no vendor holds a durable
advantage. Second, that the operational surface of a dedicated store is the real cost and
it is paid whether or not the workload grows, which is the argument in 5.14 applied to a
storage layer. Third, that the corpora most teams actually hold are small enough that
brute-force or lightly indexed search inside an existing database is sufficient, and that
the category was being sized by the largest deployments rather than the median
one.
## 3. What the paper predicted
That within two years, most production retrieval workloads would be served by vector
extensions to databases teams already ran, and the standalone category would consolidate
to a small number of vendors serving genuinely large deployments.
## 4. What was already weak at publication
The paper's own hedge was that a sufficiently large corpus with strict latency targets
might justify a specialised store. That hedge was stated in one sentence and not
developed, which in hindsight was where the whole argument was.
## 5. What actually failed
This section was added at retraction. It is not struck through, because it is the only
part of this page that is currently believed.
**5.1. The index was not a commodity.** The paper treated approximate nearest neighbour
search as a solved algorithmic problem and therefore as undifferentiated. What matters in
production is not the core algorithm but the properties around it: filtered search that
stays accurate when a predicate removes most of the corpus, incremental index maintenance
under continuous writes, quantisation that trades memory for recall in a controllable way,
and predictable behaviour at the recall and latency point a product actually needs. These
are engineering properties, they differ substantially between implementations, and the
paper dismissed the entire category of them in a subordinate clause.
**5.2. The operational story matured faster than predicted.** The argument rested on a
second stateful system being expensive to run. Managed offerings, sensible defaults and
better operational tooling arrived inside the prediction window and reduced that cost
enough to change the decision.
**5.3. The prediction was directionally right and useless.** General-purpose databases did
gain credible vector capability, which is the thing the paper said would happen. It did
not follow that the specialised systems were a fad, and the paper's confidence came from
the first observation while its conclusion depended on the second.
The paper said
What was true
The algorithm is published, so the product is undifferentiated
The algorithm is published. Filtered search accuracy, incremental maintenance and quantisation behaviour are not, and they are what a deployment lives or dies on.
A second stateful system is too expensive to operate
It was, at the time of writing, and stopped being so inside the prediction window.
Therefore the category disappears
General-purpose databases did gain the capability. The category did not disappear. The conclusion never followed from the premises.
## 6. What survived, stated narrowly
For corpora below roughly the size where index structure starts dominating latency, a
vector extension to a database the team already operates is usually the right first
choice, and the reasons are the ordinary ones in 5.14: one fewer system to back up,
upgrade and page about. That is a much smaller claim than the title, it is not
interesting, and it is what I should have written.
The larger lesson is recorded in erratum 7.2 and shapes 5.8: a confident prediction about
which technology category will disappear is a bet on a market, and my evidence was about
an algorithm. Those are different things, and the confidence value I published at the time
(0.65) was not low enough to reflect that I had substituted one for the other.
--- 5.11 What a Postmortem Owes You ---
url: https://mosthofaimran.com/papers/postmortem-owes-you/
state: holding
confidence: 0.9
revised: 2026-08-14
retires:
- An organisation publishing narrative-only postmortems, naming no decision and no owner, that nonetheless shows a falling rate of repeat incidents in the same subsystem over eighteen months.
- Evidence that requiring a named decision measurably suppresses incident reporting, so that the cost in disclosure exceeds the gain in correction.
- A study showing that action items with an owner and a verification date are completed at the same rate as those without, which would remove the mechanism this paper rests on.
Abstract. The product of a postmortem is a decision. Everything
else in the document (the timeline, the graphs, the contributing factors) exists to make
that decision legible and to let a reader disagree with it. A postmortem that names no
decision has recorded the weather. It reads as diligence, it costs several engineer-days
to produce, and the same incident recurs because nothing changed that a future engineer
can trip over. Confidence 0.90. Section 5 is the part I cannot resolve: a rule that demands a
decision will get some manufactured ones.
## 1. The claim
Ask of any postmortem: what is now different? Not "what did we learn", which is
unfalsifiable, and not "what will we do", which is a forecast. What is different, today,
in a way another engineer would notice without reading this document.
If the honest answer is nothing, the incident is unresolved regardless of how good the
write-up is. This is uncomfortable because the write-up is often genuinely excellent.
Timeline theatre is the most convincing artefact in engineering: precise, chronological,
full of real detail, and structurally incapable of changing anything.
## 2. Three failure modes
**2.1. The weather report.** A minute-by-minute account, accurate throughout, ending in a
paragraph of reflection. It documents that a thing happened. It commits to nothing. It
is the most common form and the hardest to criticise, because every individual sentence
is true.
**2.2. Root-cause singularity.** The search terminates at the first satisfying
explanation, usually the last change before the alert. Incidents in systems of any size
are conjunctions: a latent defect, a configuration drift, a missing signal, and a human
decision made with the information available at the time. Stopping at one of the four
means the other three are still armed.
**2.3. Action items with no owner, no date and no test.** "Improve monitoring." "Consider
adding a circuit breaker." An item that cannot fail cannot be completed. Six months
later the list is still open and nobody can say whether that matters.
incident
|
v
narrative -------------> archive (weather report)
|
+--> contributing factors
|
+--> decision
|
+--> owner
+--> verification date
|
v
something a future
engineer trips over
Figure 1. The upper path is the common one. Only the lower path
changes the probability of recurrence.
## 3. What the document owes the reader
Four obligations. They are ordered by how often they are skipped.
§
Obligation
1
The decision that changed. Stated as a single sentence a reader can disagree with. "We now retry at the gateway only, and the SDK returns errors." If no decision changed, say that explicitly and say why, which is a legitimate and much rarer outcome than the silence suggests.
2
The signal that would have caught it earlier, and whether it exists now. Every incident has a moment where the system knew and nobody was told. Name the metric, the threshold and the destination. If the answer is that no such signal is practical, that is a finding.
3
The person who can veto the fix. Not the owner of the action item. The person whose objection would stop it. Naming them converts a silent stall into a visible disagreement, and disagreements can be resolved.
4
The date the fix is verified in production. Verified, not merged. A fix that has not been exercised against the failure mode is a hypothesis with a commit hash.
## 4. The part that is culturally expensive
A postmortem should name the moment of the missed decision, including where it was
missed by the reviewer rather than the author. This is where blamelessness is most often
misapplied. Blamelessness means the consequence to the individual is zero. It does not
mean the record is vague. A document that will not say "this was approved in review, and
the review did not ask about the retry budget" has removed the only detail from which
the review process could learn.
Erratum 7.3 on this site is that exact case, written about me, and it is the reason I
hold this claim at 0.90 rather than lower. I have watched the vague version fail and the
specific version work, in the same organisation, eight months apart.
Named decisionrequired
Signal + thresholdrequired
Veto holderrequired
Verify daterequired
Minute-by-minute timelineoptional
Figure 2. The inversion this paper asks for. The optional row is
the one most templates make mandatory.
## 5. The strongest objection
Demanding a decision produces manufactured decisions. This is the
objection I cannot fully answer. Under a rule that every postmortem must name a change,
teams will name a change, and some fraction of those will be theatre: a lint rule, an
extra alert nobody will act on, a runbook paragraph. That is worse than honesty, because
it consumes the review budget and creates the appearance of correction. The version of
the rule I actually believe in is that the document must answer the question, and that
"nothing changed, and here is why" must be an acceptable answer that a senior person is
willing to sign. Whether that survives contact with a organisation under audit pressure,
I do not know. There is also a second objection with force: for genuinely novel failures
in an immature system, the correct output really is understanding, and the decision
follows a quarter later.
## 6. What this paper does not claim
Timelines are not useless. They are the evidence the decision rests on, and a decision
without them is an assertion. Nothing here argues for shorter documents, or for writing
one after every incident. The four obligations in Section 3 are not sufficient either. My
claim about them is weaker than it may read: a document missing any of the four has, in
my experience, failed to change the system it describes, which is not the same as saying
that a document containing all four succeeds.
--- 5.12 Your Service Boundaries Are an Org Chart ---
url: https://mosthofaimran.com/papers/service-boundaries-org-chart/
state: holding
confidence: 0.85
revised: 2026-08-14
retires:
- A system of comparable size whose service graph remained materially unchanged across a reporting-line reorganisation, sustained for four quarters, with no deliberate effort to hold the architecture in place.
- Evidence that distributed and asynchronous working has flattened the communication cost gradient enough that team boundaries no longer predict interface boundaries.
- A demonstration that the correlation runs the other way in practice, with organisations reliably reshaping their reporting lines to match an architecture chosen first, at a rate high enough to make the inverse manoeuvre in Section 4 the normal case rather than the rare one.
Abstract. Conway's law is usually quoted as an observation about
other people's systems. It is more useful as a design constraint with a predictable
timescale. Across three reorganisations I watched the service graph converge on the
reporting graph within roughly two quarters, in every case, regardless of what the
architecture documents said. The practical consequence is that a boundary you want must
be paid for in organisational structure, not in diagrams. Confidence 0.85. Three cases in
one company is not a sample, and the causal direction is not settled. Both are Section 5.
## 1. The claim
An interface hardens where communication is expensive. That is the mechanism in one
sentence, and everything else follows from it.
Inside a team, changing a function signature costs a conversation. Across a team
boundary it costs a ticket, a sprint boundary, a compatibility window and, if the teams
report to different managers, a negotiation about priority. Engineers are efficient. They
route around expense. So the seams in the codebase migrate, over months, to sit exactly
where the organisational expense is, and the architecture document becomes a description
of a system that no longer exists.
The observation is Conway's. What I am adding is that the timescale is short enough to be
useful, and that the direction of causation is asymmetric in a way that gives you a
lever.
## 2. What the convergence looks like
The pattern is not that services get renamed. It is subtler and it shows up in four
places before it shows up in the deployment topology.
Where it shows first
Symptom
Shared modules
A library owned by two teams grows a seam down the middle. Both halves are still in one repository, both are still deployed together, and no change ever crosses the seam.
API versioning
Endpoints crossing a team boundary acquire versions and deprecation policies. Endpoints inside one team keep changing in place, and nothing breaks.
Data
A table two teams write to becomes a table one team writes to and the other reads through a view, then through an event, then through a copy.
On-call
The rotation splits before the service does. The rotation boundary is the most honest architecture diagram an organisation produces, because it is the one with consequences attached.
quarter 0 reporting lines
A ---- B C
architecture [ ingest -- enrich -- serve ]
one service, three modules
quarter 2 reporting lines
A B ---- C
architecture [ ingest ] -> [ enrich -- serve ]
seam appeared where A left
Figure 1. The reorganisation happened in quarter 0. The seam was
visible in the code by quarter 2, and nobody proposed it in a document.
## 3. Why the architecture document loses
Three reasons, and none of them involve anyone behaving badly.
**3.1. The document has no enforcement surface.** A boundary that is not enforced by a
compiler, a repository permission, a deployment unit or an on-call rotation is a
suggestion. Suggestions decay at the rate of staff turnover.
**3.2. Local incentives are correct and global ones are diffuse.** An engineer avoiding a
cross-team negotiation to ship this week is making the right call for this week. The
architecture erodes one correct local decision at a time, which is why it is invisible
in review.
**3.3. Reorganisations are faster than refactors.** A reporting line changes in an
afternoon. A service boundary changes over two quarters. The organisation will always be
ahead, so the code is always converging on a target that has already moved.
## 4. Using it rather than lamenting it
The lever is that the causation is asymmetric. Organisation shapes architecture reliably
and quickly. Architecture shapes organisation weakly and slowly. So the manoeuvre is to
choose the boundary you want and then move the people, which is a management action
rather than an engineering one.
- **Before designing the service graph, draw the team graph you can actually get.** If
you cannot get it, design for the one you have. A boundary you cannot staff is a
boundary you will maintain by hand until you stop.
- **Never let two teams own one deployment unit.** It will grow a seam anyway, and the
seam will be placed by expedience rather than by design. Split it deliberately or merge
the ownership.
- **Watch the on-call rotation as the leading indicator.** When someone proposes
splitting a rotation, the service split is roughly two quarters away whether or not
anyone has written it down.
- **Price a desired boundary in headcount.** "We want ingest isolated" translates to "we
need someone to own ingest". If that role does not exist, the isolation will not
survive its first deadline.
- **When a reorganisation is announced, schedule the architecture review inside the same
quarter.** The drift is going to happen. The choice is whether it is designed or
discovered.
Reporting line changedays
Rotation splitweeks
Interface hardens~1 quarter
Deployment unit splits~2 quarters
Figure 2. The sequence I have observed three times. The ordering is
the claim; the durations are approximate and drawn from three cases, which is
not a sample.
## 5. The strongest objection
Three reorganisations in one organisational culture is an anecdote,
and the direction of causation is not established. Both halves of that are fair. My
three cases share an employer, a market and a hiring pipeline, so what I may have
observed is a property of one company rather than of organisations. And the causal claim
is genuinely underdetermined: it is equally consistent with the evidence that managers
reorganise in anticipation of architectural pressure they can already see, which
would make the reporting change a symptom rather than a cause and would invert the advice
in Section 4 entirely. I hold 0.85 because the predictive value has been high for me,
not because the mechanism is settled. A well-constructed study across several
organisations would move this number in either direction, and I would rather have it
moved than defended.
## 6. What this paper does not claim
Boundaries should not follow the org chart as a matter of preference. Sometimes the right
architecture cuts against it, and then the work is to change the org chart, which is the
whole point of Section 4. The stronger argument that microservices are themselves an
organisational artefact is a separate paper and I am not making it here.
The effect is also not inevitable. Resisting it works. It costs continuous effort, and
the failure I keep watching is that the effort is assumed rather than budgeted.
--- 5.13 The Dashboard Nobody Opens ---
url: https://mosthofaimran.com/papers/dashboard-nobody-opens/
state: holding
confidence: 0.75
revised: 2026-08-14
retires:
- A team whose dashboards are built from emitted signals rather than from operator questions, and whose new on-call engineers nonetheless answer the five triage questions in Section 3 within ninety seconds, without assistance, on an incident they have not seen before.
- Evidence that question-driven dashboards measurably slow diagnosis of novel failure modes, so that the breadth they discard costs more than the speed they buy.
- Query-side tooling that makes ad-hoc exploration fast enough that pre-built views stop mattering, at which point this paper is about a tool generation rather than about a practice.
Abstract. Most dashboards are assembled from the signals that
were easy to emit rather than from the questions an operator asks under pressure. The
result is a wall of accurate panels that answers none of the five things anyone actually
needs to know at 03:00, and which is therefore not opened during the incident it was
built for. The fix is to invert the construction order: enumerate the questions first,
build one view per question, and delete anything that answers none of them.
Confidence 0.75. Section 5 carries a cost I cannot price: views built from
questions over-fit the failures already survived.
## 1. The claim
Observability is usually built bottom-up. A library emits what it can, a platform team
collects it, and a dashboard is assembled from what arrived. Every panel on that
dashboard is true. The collection is complete, the queries are correct, the alerting is
wired. And during the incident, the on-call engineer opens it, looks at it for four
seconds, and goes to the logs.
That four seconds is the measurement that matters and nobody records it. A dashboard's
value is not the number of signals it displays. It is the number of triage questions it
answers without a follow-up query.
## 2. Why bottom-up construction fails
The signals that are cheapest to emit are properties of components. The questions that
matter are properties of the interaction between components and users. These are
different objects, and no amount of correct aggregation turns one into the other.
emitted (cheap) asked (expensive)
cpu, memory, disk is it me or them?
request rate all users or some?
error count did it start with a deploy?
p50 / p95 / p99 is it getting worse?
queue depth what breaks next?
Figure 1. The left column is what most dashboards contain. The right
column is what the person looking at them is trying to find out.
The right-hand column is answerable from the left-hand column, and that is exactly the
problem: it is answerable by an engineer doing arithmetic and correlation in their head,
under time pressure, with a pager going off. The dashboard has delegated the hard part.
## 3. The five questions
These are the questions I have watched people ask, in this order, in every triage I have
been part of. Any dashboard that does not answer them is decoration.
§
Question
What the panel must show
1
Is it us or a dependency?
Error rate split by originating layer, with dependency errors attributed to the dependency rather than counted as ours.
2
Everyone or some?
The same failure rate broken by tenant, region and client version, on one view. This is the panel that most often does not exist, because the dimension was never emitted.
3
Did it start with a change?
Deploy and configuration-change markers on the same time axis as the failure. A separate deploy dashboard does not count; correlation across two browser tabs is the work being avoided.
4
Is it getting worse?
Rate of change, not level. A flat 4% error rate and a 4% rate doubling every two minutes look identical on a gauge and require opposite responses.
5
What fails next?
Saturation of the things that will queue: pools, buffers, budgets. Including the retry budget, which is the signal from 5.6 that almost nobody emits.
## 4. The construction rule, and the test
**4.1. One view per question, named after the question.** Not "Service Health". The panel
title is the interrogative sentence. This reads as unserious and it is the single change
with the largest effect, because it makes an unanswerable question visibly unanswered
rather than quietly absent.
**4.2. If a signal answers no question, it is not on a dashboard.** Keep emitting it,
keep it queryable, keep it out of the view. Storage is cheap and attention is not.
**4.3. Instrument the dashboards themselves.** Most observability stacks can report which
views were opened and when. A dashboard with no opens in ninety days is either wrong or
redundant, and in both cases deleting it is an improvement. This is the only part of the
paper with a direct measurement attached, and it is the part teams resist most.
**4.4. The ninety-second test.** Take an engineer who has been on-call for under a month.
Give them a real past incident. They should answer all five questions in ninety seconds
without asking anyone. If they cannot, the gap is a work item with an owner, and the
dashboard is the deliverable, not the diagnosis.
Q1 us or themcommon
Q2 all or somerare
Q3 change markerssometimes
Q4 rate of changerare
Q5 saturationpartial
Figure 2. How often each question is already answered, ranked from
the systems I have reviewed. A ranking from memory, not a survey.
## 5. The strongest objection
Question-driven dashboards over-fit the failures you have already
had. This is the real cost and I do not have a clean answer to it. The five questions
are derived from incidents I lived through, which means a view built to answer them is
optimised for a distribution of failures drawn from the past. The expensive incidents are
the novel ones, and novel incidents are diagnosed by exploration: broad, unfiltered,
looking at things nobody thought to put on a panel. A practice that deletes unopened
views is deleting exactly the breadth that exploration needs. The reconciliation I use is
that exploration should happen in the query interface rather than on dashboards, and that
dashboards are for triage only. That splits the tools cleanly in theory. In practice
teams with weak query tooling use dashboards for both, and for them this paper's advice
would make things worse. That is most of the gap from 0.75 to 0.9.
## 6. What this paper does not claim
Component metrics are necessary. Questions 1 and 5 are answered from them, and a system
that does not emit them cannot answer either. The five questions are not a complete set;
they are what has been sufficient for first-response triage in the systems I have run.
Dashboards do not cause slow incident response. The claim is smaller and more specific: a
dashboard assembled from what was easy to emit is answering a question nobody asked, and
that cost stays invisible because almost nobody measures whether it was opened.
--- 5.14 Kubernetes for a Bicycle ---
url: https://mosthofaimran.com/papers/kubernetes-for-a-bicycle/
state: holding
confidence: 0.7
revised: 2026-08-14
retires:
- A team of five or fewer engineers, with no dedicated platform role, running a full orchestration stack for eighteen months while spending less total time on the platform than on the product, measured rather than recalled.
- Managed orchestration reaching a point where the fixed costs listed in Section 3 are genuinely absorbed by the provider, including upgrade cadence, network policy and on-call knowledge, at which point the break-even in Section 4 moves far enough to invert the advice.
- Evidence that starting simple and migrating later costs more in aggregate than starting heavy, which is the inverse of the assumption this paper rests on and the one I would most like to see tested.
Abstract. Operational weight has a fixed cost and a variable cost.
The fixed cost is paid whether or not the load arrives, it is mostly not the cluster, and
it is routinely estimated at a fraction of its real size. I have made this mistake twice,
in both cases having costed the infrastructure and not the practice that has to surround
it. The claim is not that orchestration is wrong. It is that the sizing question is
"how many operators do we have" rather than "how many nodes do we need".
Confidence 0.70. Section 5 has two objections I cannot weigh against each other,
and I have been burned by the second one as well.
## 1. The claim
Choosing an operational model is choosing a fixed monthly cost in engineer-hours. That
cost is incurred on the quiet weeks as well as the busy ones, and it is paid by the same
people who are supposed to be building the product.
The bicycle in the title is the load. The failure is not that the machinery is bad
machinery. It is that the machinery has a minimum operating crew, and a team that cannot
staff that crew ends up operating it badly, which is worse than operating something
smaller well.
Both times I got this wrong I had done the arithmetic. Both times the arithmetic was
about the cluster. The cluster was never the expensive part.
## 2. Fixed and variable, drawn honestly
effort
^
| heavy stack
| ------------------------------ fixed floor
| /
| /
| light stack /
| ____________/
+--------------------------------> load
^
break-even, which is
further right than it looks
Figure 1. The heavy stack wins eventually. The question is whether
your load reaches the crossing point before your team runs out of the hours the
floor consumes.
The shape is uncontroversial. The mistake is in placing the floor, and the floor is
placed by what is on the list in Section 3 rather than by the orchestrator itself.
## 3. What the fixed cost is actually made of
Cost
Why it is underestimated
Upgrade cadence
The platform has a support window measured in months. Somebody owns that treadmill permanently, and it does not scale down when the product is quiet.
Network policy and identity
The default posture is usually open, so a real deployment needs policy, service identity and secret distribution designed rather than adopted. This is where most of the first quarter goes.
The build and release path
Registries, image provenance, signing, promotion between environments. All defensible, all work, none of it visible in the decision that started it.
Observability of the platform itself
You now have two systems to watch: the product, and the thing running it. Both page.
On-call knowledge
The largest and least tracked. Every engineer in the rotation needs a working model of the platform's failure modes, which is a training cost paid per person and again on every hire.
Bus factor on the platform
In a small team this is usually one person, and the fixed cost is invisible until that person takes leave.
Cluster itselfcosted
Release pathpartly
Policy and identitymissed
On-call knowledgemissed
Figure 2. What I costed against what it cost, both times. Drawn from
two cases in hindsight, which is not a sample.
## 4. The ladder, and the triggers for climbing it
The useful discipline is not choosing correctly on day one. It is naming, in advance, the
observation that will move you up a rung. Without a named trigger the decision is made by
whoever is most enthusiastic in the room.
Rung
Move up when
One machine, one artifact, a service manager
A single machine's failure becomes an unacceptable outage, or deploys need to be zero-downtime.
Two or three machines, a load balancer, deploys by script
You are hand-placing more than about ten distinct workloads, or bin-packing has become a spreadsheet.
Managed container runtime
You need scheduling policy the platform cannot express: affinity, priority classes, custom autoscaling, per-tenant isolation.
Full orchestration
You have a platform owner. Not a volunteer. A role.
The last trigger is the whole paper compressed into one line. If the answer to "who owns
the platform" is a name plus the word "also", the rung is too high.
## 5. The strongest objection
The ecosystem argument, and it is strong. Orchestration is the
industry default. Choosing something smaller means every vendor integration, every hire,
every piece of tooling and every published runbook is written for a platform you are not
running, and that ongoing translation cost is real and compounding. It also affects
recruitment in a way that is difficult to price and easy to dismiss. Against that, my
counter-argument is only that the translation cost is visible while the operational floor
is not, and that people systematically overweight visible costs. I believe that, but I
cannot demonstrate it, and it is most of the distance from 0.70 to a higher number.
There is a second objection with force: premature simplicity is also a failure mode, and
migrating a running system up a rung under load is more expensive than starting one rung
high. I have watched that go badly too. Nobody has given me a principled way to weigh the
two regrets against each other, so what follows Section 4 is advice and not analysis.
## 6. What this paper does not claim
Orchestration is not over-engineering. At sufficient scale it is the cheapest option
available and the fixed cost is recovered without anyone noticing. Managed offerings do
help; they absorb less of the list in Section 3 than their marketing implies, which is a
complaint about the marketing rather than about the product. The ladder in Section 4 is
one sensible ordering and not the only one.
Nor have I fully learned this. Two occurrences is a pattern, not a cure, and I would not
bet against a third.