Mosthofa Imran: Systems, Arguments,
and Known Failure Modes
Status of This Memo
Abstract
Eleven years of production engineering. I design multi-tenant agentic systems that ship from a single build into public cloud and into air-gapped data centres inside regulated banks: active-active across two sites, tens of millions of events a day, recovery objectives measured in minutes because the alternative is a conversation with a regulator.
I also write. The running argument is that this industry has learned to manufacture the feeling of competence faster than it can produce the thing itself, and that a good deal of what is sold as developer experience is friction relocated to a worse hour. Section 5 states those claims in a form that can be attacked, and Section 7 lists the ones that were.
Table of Contents
1. Introduction #
Most personal sites are a pitch. This one is a specification, because a specification is the only format I trust to survive contact with a reader who knows more than I do. A pitch invites agreement. A specification invites a diff.
Two kinds of object live here, and they are held to the same standard. Implementations are systems I built; they carry throughput, latency and a named failure. Position papers are claims I make; they carry a confidence value and the evidence that would retire them. Anything that carries neither is marketing, and marketing belongs on a different domain.
The order is deliberate. Systems come before opinions because the systems are what paid for the opinions. The retraction log comes after both because it is the only section that proves either was arrived at honestly.
2. Conventions and Terminology #
2.1. Requirements Language
The key words MUST, MUST NOT, SHOULD, MAY and WILL NOT appear in Section 4 with their usual force [RFC2119], with one local amendment: here they describe commitments already kept under production pressure, not intentions. Where one has been broken, it is recorded in Section 7 with the incident that broke it.
2.2. Terminology
| Term | Definition |
|---|---|
| Confidence | A number in [0,1]: my credence that a claim's central mechanism still holds in five years. Published even when it is low, because a low number is information and silence is not. |
| Holding | No material objection has landed since the last revision. |
| Revising | An objection landed. The text is currently wrong in a way I have not finished repairing, and it says so at the top. |
| Retracted | The central claim failed. The text stays, struck through, with the reason and the person who found it. Nothing here is deleted. |
| Sovereign | No prompt, completion, embedding, log line or metric leaves the customer's network boundary. Enforced at network policy and verified on a schedule, not asserted in a slide. |
| Named failure | A failure mode described with its trigger, its blast radius and whether it is fixed. A system listed without one is a system I do not understand yet. |
3. Implementations #
Systems in production, with the numbers they produced. Each links to a full implementation note carrying constraints, decisions, measurements and known failure modes.
| § | System | Result | State |
|---|---|---|---|
| 3.1 | Mevrik: agentic customer experience platform Multi-tenant conversational automation. One artifact ships to public cloud and to air-gapped on-premise estates inside regulated banks. Active-active across two data centres, RTO under four minutes, tested quarterly with real failover rather than a tabletop. Rust · Python · ClickHouse · PostgreSQL · RabbitMQ · Redis · Kubernetes | 40M events/day 3 regulated tenants | Production |
| 3.2 | Sovereign LLM gateway One control plane over four model backends. Redaction runs before routing, fallback is deterministic and stamped, quotas are per tenant and per model. Tenants move between hosted and local models with a config change rather than a rewrite. Python (FastAPI) · Redis · OpenSearch · PostgreSQL · Ollama | 3.1M calls/day p99 overhead 180 ms | Production |
| 3.3 | Ingest path rewrite, PHP to Rust Identical semantics. Verified by replaying ninety days of production traffic through both implementations and diffing the outputs byte for byte before any customer saw it. Rust (tokio) · RabbitMQ · PostgreSQL · MinIO | 89% fewer nodes p99 340 ms → 11 ms | Production |
| 3.4 | Analytics migration to ClickHouse Fourteen billion rows moved under a dual-write cutover. Zero downtime, and one rollback executed cleanly at 02:40 that nobody outside the team noticed. The rollback is the part I am proud of. ClickHouse · Kafka Connect · Airflow | 14B rows query p95 9.4 s → 380 ms | Complete |
| 3.5 | Air-gapped delivery pipeline Signed, reproducible bundles carried across the gap on physical media, verified on arrival, installed by an operator who has never met us and cannot phone us. Every assumption a cloud deployment gets for free had to be paid for in cash. OCI bundles · cosign · Helm · offline registry | 6 sites 0 failed installs since 2024-09 | Production |
4. Operating Principles #
Normative language, applied to myself. Each of these cost something to learn. Section 7.3 records the one I broke.
4.1. A service MUST degrade loudly. A silent quality drop is worse than an error, because an error gets investigated and a silent drop gets absorbed by the user until they leave without telling you.
4.2. A security control MUST NOT depend on the correctness of the next hop. Redaction that runs after routing is decoration with a compliance budget.
4.3. Retries MUST carry jitter, a budget and an idempotency key. Two out of three builds a very reliable metronome for your own outage.
4.4. On-premise and cloud SHOULD ship from one artifact. Two build paths means the air-gapped customer runs the version nobody tests, and they find out first.
4.5. Every abstraction that removes friction MUST document where the friction went. It did not evaporate. It moved, usually into the incident, usually at 03:00.
4.6. A fence in the codebase SHOULD be read before it is removed. Chesterton's fence has a git blame, and the person who put it there left a commit message.
4.7. Capacity plans MUST be stated as a failure point, not a target. "Handles 10k rps" tells you nothing. "Falls over at 12k rps, and here is what it does on the way down" tells you everything.
4.8. I WILL NOT present an architecture without its failure mode. Not in a talk, not in a proposal, not in this document. A design shown only in its best light is a jet engine bench-tested at sea level and sold as a mountain crossing.
5. Position Papers #
Claims, with confidence values and revision dates. Full index at Section 5, complete. Feed at /feed.xml.
| § | Claim | Conf. | State | Revised |
|---|---|---|---|---|
| 5.1 | Competence Porn Watching a skilled person work occupies the same reward slot as being one, and the industry has industrialised the confusion. | 0.80 | Holding | 2026-06-11 |
| 5.2 | Vibe Coding and the IKEA Effect We overvalue systems assembled from parts we never read. Assembly produces the sensation of understanding without the substance. | 0.75 | Holding | 2026-05-02 |
| 5.3 | Algorithmic Homophily Your timeline is a cache with no invalidation policy. It returns the priors you arrived with, warmed. | 0.60 | Revising | 2026-08-02 |
| 5.4 | The Easy Button Tax Removed friction is relocated friction. The invoice arrives during the incident, denominated in hours you do not have. | 0.85 | Holding | 2026-04-20 |
| 5.5 | On-Premise Is Not a Downgrade Sovereignty is an architectural constraint, not a punishment. Design for it early and it stops costing you. | 0.90 | Holding | 2026-03-19 |
| 5.6 | The Retry Storm You Built On Purpose Exponential backoff without jitter synchronises your clients. You shipped the accelerant, then wrote a postmortem about the fire. | 0.95 | Holding | 2026-07-28 |
| 5.7 | Chesterton's Fence Has a Git Blame The argument for reading history before deleting it, aimed at the specific case where the history is one line and a date. | 0.80 | Holding | 2026-02-14 |
| 5.8 | RAG Is a Search Problem in a Trench Coat The embedding model is rarely the bottleneck. The chunking strategy is an information retrieval decision made in an afternoon and never revisited. | 0.70 | Holding | 2026-01-30 |
| 5.9 | The Ship of Theseus Passes Its Integration Tests On strangler-fig migrations, and the moment where the new system is technically complete and nobody can say when it became itself. | 0.55 | Draft | 2026-08-09 |
| 5.10 | n/a | Retracted | 2025-11-14 |
6. Curriculum Vitae (access controlled) #
6.1. Rationale
Sections 3 through 5 are a more honest record than any résumé: they carry numbers, named failures and revision dates, and they can be checked against people who were in the room. The document itself carries client names, reference contacts and a phone number that belong to other people's privacy as much as to my convenience, so it is issued on a signed link rather than served as a page.
This is not scarcity theatre. If you would rather not leave an address, write to me and I will send the file back by hand. That path is listed first in 6.5 because it is the one I prefer.
6.2. Request
Not built yet. The endpoint ships with Section 6 and this form is inert until it does. Principle 4.1 says a service must degrade loudly, so it says so here rather than accepting an address and dropping it. Until then use the path 6.5 lists first anyway: write to imran@mosthofaimran.com and say what the role is.
Returns 202 Accepted and mails a signed URL. Time to live 86400 seconds,
single use, X-Robots-Tag: noindex. Rate limited to three requests per address per day and
sixty per source network per hour. Your address is written to the access log described in 6.4 and to
nothing else. It is never added to a list, never sold, never used to start a sequence, and it is
deleted the same day you ask.
6.3. Protocol
Requester mosthofaimran.com Token store
| | |
| POST /api/cv {email} | |
|--------------------------->| |
| | PUT token, ttl=86400s |
| |--------------------------->|
| 202 Accepted | |
|<---------------------------| |
| | |
| ...mail delivered, signed URL enclosed... |
| | |
| GET /cv/{token} | |
|--------------------------->| CONSUME token |
| |--------------------------->|
| 200 application/pdf | ok, burn |
|<---------------------------|<---------------------------|
| | |
| GET /cv/{token} (again) | |
|--------------------------->| CONSUME token |
| |--------------------------->|
| 410 Gone | already burned |
|<---------------------------|<---------------------------|
6.4. What is logged
| Field | Retention | Purpose |
|---|---|---|
| Email address | Until deletion requested | Issue the link, and revoke it if you ask |
| Issued and redeemed timestamps | 12 months | Detect token sharing, which tells me a recruiter forwarded it |
| Source network, truncated to /24 | 30 days | Rate limiting only |
| User agent | Not stored | There is no purpose, so it is not collected |
6.5. Alternatives
| Preference | Path |
|---|---|
| Preferred | Write to imran@mosthofaimran.com and say what the role is. I reply with the file and a paragraph about whether I think it is a fit, including when I think it is not. |
| Fine | Use the form in 6.2. |
| Also fine | Do not request it at all. Sections 3 through 5 will tell you more about how I work than two pages of reverse-chronological employment will. |
7. Errata and Retractions #
This section is load bearing. A body of opinion with no retraction log records positions rather than learning. Corrections are credited to the person who sent them, and published to /errata.xml so you can subscribe to my mistakes specifically.
| § | Correction | Date |
|---|---|---|
| 7.1 | Paper 5.3 downgraded from 0.80 to 0.60. Two readers demonstrated the effect reproduces in private mailing lists with no ranking algorithm present, which breaks the mechanism I proposed. Section 3 of that paper is being rewritten and currently says so at the top. Reported by A. Rahman and K. Osei. | 2026-08-02 |
| 7.2 | Paper 5.10 retracted in full. The central claim failed. Retrieval quality at scale turned out to depend on index properties I had dismissed, and the operational story matured faster than I predicted. What I was actually right about was narrower and less quotable. Original text preserved, struck through, with this note attached at the head. | 2025-11-14 |
| 7.3 | Principle 4.3 was violated in production, by me. The June 2025 ingest incident traced to a retry path with backoff and no budget, in code I reviewed and approved. Ninety-four minutes of degraded intake. Postmortem published unredacted, including the review comment where I did not ask the question. | 2025-06-30 |
| 7.4 | Figure 3 of the ClickHouse note had the axis mislabelled. Milliseconds, not seconds. The conclusion was unaffected but the chart was flattering by a factor of a thousand, which is exactly the kind of error that should be published rather than quietly swapped. Reported by S. Datta. | 2025-02-11 |
8. Publishing Model #
Papers are written in Markdown, held in git, signed, and rendered to static HTML at build time. There is no database in the reading path and no third-party script on the page. A paper is a file; its history is the commit log; its authority is a detached signature.
| Stage | Mechanism |
|---|---|
| Authoring | Markdown with front matter carrying confidence, state, published, revised, retires (the falsification conditions) and errata. The schema is validated at build; a paper without falsification conditions fails the build rather than shipping quietly. |
| Revision | Papers are edited in place. Every substantive edit appends to the paper's history table and, if it changes the claim, an entry in Section 7. The URL never changes and nothing is deleted. |
| Rendering | Static site generation. Every section of this document is a real page with a real URL; the client-side routing in this prototype is a convenience, not the architecture. |
| Signing | Each published paper gets a detached OpenPGP signature at /papers/<slug>.asc over the canonical Markdown, so a quote can be checked against what was actually written. |
| Distribution | Atom feeds, plain text mirrors, and an llms.txt for machine readers. No newsletter, no popup, no interstitial asking you to subscribe before you have read a sentence. |
9. Machine Readability, Feeds and Signatures #
9.1. Feeds
| Resource | Contents |
|---|---|
| /feed.xml | Atom. New position papers and implementation notes. |
| /errata.xml | Atom. Corrections, downgrades and retractions only. Subscribe to this one if you want to know whether I can be trusted, rather than what I think this week. |
| /revisions.xml | Atom. Every substantive edit to an existing paper, with a diff summary. |
| /papers/index.json | The full paper index with confidence, state and dates, as JSON. Stable schema, versioned. |
9.2. Machine readers
A growing share of this document's readers are not people. That is fine, and it is planned for rather
than resented. The site publishes an llms.txt so a model retrieving it gets structure
instead of a scraped navigation bar, and a full-text mirror so nobody has to render JavaScript to read
prose that was written in Markdown.
One request, stated plainly because a robots directive cannot express it: if you quote a claim from Section 5, carry its confidence value with it. A 0.60 claim repeated as fact is no longer my claim.
| Resource | Contents |
|---|---|
| /llms.txt | Structured summary of the site, its sections and its conventions, including what the confidence values mean. |
| /llms-full.txt | Every paper and implementation note as one plain text document, with front matter preserved. |
| /robots.txt | Indexing is welcome. /cv/ and /api/ are disallowed and also enforced server side, because a robots file is a request and not a control. |
9.3. Signatures
Anything worth attributing to me is signed. Verify before you trust, including when the thing you are verifying is flattering.
$ curl -sO https://mosthofaimran.com/papers/competence-porn.md $ curl -sO https://mosthofaimran.com/papers/competence-porn.md.asc $ gpg --recv-keys 0xREPLACE_WITH_REAL_KEY_ID $ gpg --verify competence-porn.md.asc competence-porn.md
| Key | Fingerprint |
|---|---|
| OpenPGP | REPL ACEM EWIT HREA LFIN GERP RINT 0000 0000 0000 · /pgp.asc |
| SSH (signing) | SHA256:replace-with-real-ssh-signing-key-fingerprint |
| Commits | Every commit in the site repository is signed. An unsigned commit in that repository is not from me. |
10. Security and Privacy Considerations #
Every RFC has this section and most of them are honest, which is more than can be said for the average privacy policy. Here is the whole of it.
| Property | Statement |
|---|---|
| Third-party requests | None. No fonts, no analytics, no tag manager, no embedded video, no comment widget. The page you are reading came from one origin. |
| Cookies | None are set. There is therefore no consent banner, which is the correct way to not need one. |
| Server logs | Source network truncated to /24, path, status, timestamp. Retained 30 days. Read roughly monthly. |
| Résumé tokens | Single use, 86400 second lifetime, revocable, never guessable. See 6.3 and 6.4. |
| Transport | HTTPS only, HSTS with preload, a content security policy that forbids inline and remote script in production, and no mixed content anywhere. |
| Data subject requests | Mail me. Deletion happens the same day, and I will tell you what was held before it goes. |
| Disclosure | Found something broken in the gate? /.well-known/security.txt. I will credit you in Section 7 unless you would rather I did not. |
11. IANA Considerations #
This document has no IANA actions.
It is included because every RFC has one, and because a document that only contains the sections it needs has quietly told you it was written for the reader's convenience rather than the form's integrity.
12. Acknowledgements #
To the operations engineer in Chattogram who took my call at 02:40 during the ClickHouse cutover and talked me out of the clever fix, and to everyone who has sent a correction that made it into Section 7. Being told you are wrong by someone who read carefully is the fastest form of professional development available, and it is free.
13. References #
13.1. Normative References
| Tag | Reference |
|---|---|
| [RFC2119] | Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. |
| [RFC1925] | Callon, R., "The Twelve Networking Truths", RFC 1925, April 1996. Truth (3) is load bearing in most architecture reviews I have attended. |
| [RFC8890] | Nottingham, M., "The Internet is for End Users", RFC 8890, August 2020. |
13.2. Informative References
| Tag | Reference |
|---|---|
| [PERROW] | Perrow, C., "Normal Accidents: Living with High-Risk Technologies", 1984. The book that explains why your postmortem keeps finding a root cause that is not one. |
| [COOK] | Cook, R., "How Complex Systems Fail", 1998. Four pages. Read it twice a year. |
| [SCOTT] | Scott, J.C., "Seeing Like a State", 1998. Every argument about data sovereignty is downstream of this, whether or not the people having it know. |
| [CONWAY] | Conway, M., "How Do Committees Invent?", 1968. Your service boundaries are an org chart with better uptime. |
| [NORMAN] | Norman, D., "The Design of Everyday Things", 1988. Applies to APIs more than to doors. |
14. Author's Address #
| Channel | Address |
|---|---|
| Name | Mosthofa Imran |
| Role | Lead Solutions Architect |
| Location | Dhaka, Bangladesh (UTC+6). Comfortable overlapping with CET mornings and US Pacific evenings. |
| imran@mosthofaimran.com. I read everything and answer most things inside two working days. | |
| Code | github.com/mosthofaimran |
| Résumé | Section 6 |
| Key | /pgp.asc, fingerprint in 9.3 |
| Not present on | Any feed-ranked platform. Paper 5.3 explains the reasoning, and holds me to it. |
Appendix A. Document History #
| Date | Change |
|---|---|
| 2026-08-13 | Section 6 completed. Sections 8 and 9 added: publishing model, feeds, machine readers, signatures. Renumbered; no gaps. |
| 2026-06-11 | Paper 5.1 revised, fourth revision. |
| 2026-01-04 | Confidence values applied retroactively to all papers published before 2026. Six of them moved. |
| 2025-08-20 | First publication. |
Appendix B. Colophon #
| Item | Detail |
|---|---|
| Typefaces | Whatever your device already has. Charter or Georgia for prose, your system monospace for everything else. Nothing is downloaded, so nothing can be blocked, and nothing phones home. |
| Build | Static HTML from Markdown. One stylesheet, no framework in the reading path. |
| Weight | One HTML file, one stylesheet, inlined at build. No script in the reading path and no third-party request of any kind. It renders on a slow connection and prints as a passable specification. |
| Hosting | Static files on a CDN. One small function for Section 6, and nothing else that can page me. |
| Licence | Prose under CC BY 4.0. Quote it, argue with it, carry the confidence value with it. |
Signature #
Detached signature over the canonical Markdown source of this document, published at /rfc0001.md.asc.