TurboHub

TurboHub was A2 Hosting’s customer control panel — where their WordPress customers managed sites, plugins, staging environments, and performance. It was built alongside WHMCS rather than on top of it, because A2’s control panel was WHMCS: a vendor billing product doing double duty as the customer experience, offering no extension points in the places that mattered most. We architected TurboHub and then led the team that built it out over the next two years.

The first commit lands on February 10, 2023. The last lands on February 4, 2025. Just under two years, roughly twenty-five people, two new repositories, and an agent daemon running on hosting servers in four datacenters. The whole architecture was designed to make WHMCS removable. It never was, and the reason is part of the story.

Building inside something you plan to outlive

The founding spec opens with an ordinary problem statement — the company was repositioning around WordPress, and the WHMCS client area was slow to customize and hard for customers to navigate — and then does something most specs don’t. It spends most of its length on decisions about what the new application is forbidden to do.

The rule everything else hangs off is that TurboHub would never query the WHMCS database directly. REST APIs only. The rationale is written down: connecting to WHMCS’s MySQL “is very easy and could make development faster. However, this will lead to us re-implementing WHMCS rather than eliminating its constraints.” Everything else follows. WHMCS stayed the source of truth. WHMCS kept billing, ordering, provisioning, authentication, and email, and had to keep working with TurboHub switched off entirely. Every third-party client got a generic wrapper A2 owned, so a vendor could be swapped without business logic noticing.

All of that existed to buy one thing: an option. The spec’s final section is titled “WHMCS Exit Plan™”, and it lays out soberly and at length what it would take to make TurboHub the system of record — users, invoices, provisioning, a migration tool, and a client-by-client cutover measured in weeks rather than days. Nobody was pretending it was cheap.

So the defining constraint was a prohibition rather than a requirement. Not “build this,” but “do not reach for the thing sitting six inches away, however much faster it would make this afternoon.” Everything below is downstream of that rule — including the places obeying it cost us, and the one place we broke it.

Three programs and a broker

turbohub-web was the customer-facing application: Laravel 10 on PHP 8.2, and by some margin the largest of the three — a full customer surface of web routes, models, services, console commands, and queued jobs. The front end inherited its Bootstrap theme wholesale from the WHMCS client area, with Inertia and Vue layered in later, which is a story of its own.

turbohub-agent ran on the hosting servers: a Laravel Zero application compiled to a single PHAR, a handful of console commands over a couple of dozen services. It had no HTTP interface and opened no inbound port. Everything it did arrived on a message queue.

The WHMCS addon is where it matters: five files, next to two applications many times its size. It exposed about a dozen actions — identify this user, list the services they may manage, resolve a cPanel username back to a service, create a DNS record, send a templated email — and nothing else. No invoices, no payments, no orders, no provisioning.

That thinness is the architecture, not an accident. The spec required WHMCS to keep working with TurboHub down. The corollary nobody wrote down is that the broker between them had to stay small enough to be disposable in both directions. A fat addon would have taught WHMCS to depend on TurboHub, the exact coupling the exit plan existed to prevent. The place scope creep would have been easiest is the place it would have cost the most.

A control panel with no users

TurboHub has no users table. No passwords, no login form, no password reset, no sessions of its own. Authentication is one middleware class, and what it does is ask WHMCS.

A request arrives carrying the customer’s WHMCS cookies. The middleware hashes the session cookie, and if that hash doesn’t match what’s cached, it builds a cookie jar from every WHMCS cookie on the request, forwards them along with the original user agent and client IP, and asks two questions: who is this, and what may they touch. The answer comes back as a graph — the user, every client account they manage, whether they’re staff masquerading, and the permission strings WHMCS itself uses. Only then does the request reach a controller.

Caching the hash of the cookie rather than a boolean is the detail that makes this affordable. If the cookie changes, the cache misses and we re-verify immediately; if it doesn’t, we skip the round trip for two minutes. Short enough that a logout takes effect before anyone notices, long enough that we weren’t hammering WHMCS on every request.

Two minutes still isn’t instant, so invalidation is pushed rather than waited out. WHMCS hooks fire on logout, on client edits, and on every service create, terminate, suspend, and unsuspend, calling back into TurboHub to bust the affected keys. TurboHub then broadcasts a refresh over Pusher to that user’s private channel, and their open browser tab reloads itself. A support agent editing a service in WHMCS causes a customer’s TurboHub tab to redraw with correct data, in an application that has never heard of that customer.

Delegation sits on top of the same mechanism. An agency could be invited to manage a client’s services, with WHMCS’s own permission vocabulary deciding whether that meant Manage or Manage plus Billing — mirrored, not reinvented. Staff could masquerade. And internal users could override their own access tier to see the application exactly as a beta or a stable customer saw it, which mattered more than it sounds: when your app is embedded in someone else’s navigation, “works for me” and “works for a customer” are two different claims.

Between the two applications, auth was a custom bidirectional server-to-server OAuth grant, so WHMCS could call TurboHub with the same credential TurboHub used to call WHMCS. Tokens rotated every six hours, and the retry loops re-read the token inside the loop rather than outside it, so a call that started before a rotation and retried after it still succeeded. Small detail; the kind that only ever shows up at three in the morning.

The last piece is a lie. TurboHub ran on its own host, but nginx overwrote SERVER_NAME and HTTP_HOST with the WHMCS domain before handing the request to PHP-FPM, so Laravel generated URLs and set cookies as though it were part of the billing panel. Customers never saw the seam because of it. The cost came due in two places: URL-generation bugs were undetectable in development and showed up only behind the proxy, and it was the single piece of the architecture that would have had to be unwound on the day WHMCS went away, in a system built entirely to make that day cheap.

The judgment underneath all of it is that we paid an HTTP round trip on requests to avoid a SELECT we could have written in an afternoon. That round trip was the point. Swap the application underneath and this is the one subsystem that keeps working, which is the whole reason it exists.

Addressing the fleet

The spec listed two unknowns for the proof of concept. The first was authentication. The second was how to issue commands to a WordPress install on a hosting server, and the answer at the time was SaltStack. A Salt client still sits in the codebase with zero call sites — a fossil from a plan that lasted about six months. What replaced it, in August 2023, was an agent of our own and a message queue.

The queue name is the server. Communication is RPC over AMQP: the web application publishes a message carrying a reply queue and a correlation ID, blocks for up to ninety seconds, and the agent publishes the result back. The queue a message goes to is the server’s hostname, or its dedicated IP for VPS and dedicated boxes. That one choice collapses three things into a single fact — addressing a server, naming a server, and authorizing a server are the same act — so there is no routing table to keep in sync and no way to address a machine you aren’t entitled to.

Dispatch on both ends is a configuration array mapping dotted strings to class methods, resolved by reflection: wordpress.plugins.update becomes a method call on a plugin service. The agent’s route file is therefore its public API — about fifty entries covering WordPress core, config, health, maintenance, plugins, themes, security, optimization, cPanel, sites, staging, and the server itself. Adding a capability meant writing a method and adding a line, and reading one file told you everything the platform could make a server do. Nobody had to maintain a document describing the interface, because the file was the document.

Credentials arrive at boot, or not at all. The agent ships with none. On start it works out its own identity — hostname on shared servers, its own IP on dedicated ones, skipping a short list of known provisioning-image addresses because a freshly built box briefly answers as the template it was cloned from. Then it presents a bootstrap token to a central monitoring service over TLS and receives AMQP credentials scoped to itself.

The engineering is in the failure paths, which are driven entirely by response code. A 403 means the identity didn’t check out, so the agent wipes its own bootstrap token on the assumption it has been spoofed or revoked. A 404 means the server isn’t known, so it sleeps ten minutes before exiting rather than restarting into a tight loop — the comment says “so we don’t flood monitor.” A 429 exits without retrying at all, with the comment “possible denial of service attack. quit and don’t try again.” Everything else exits and lets the process supervisor try again later.

The result is a trust model you can state in one sentence: a server can only consume from the queue named after itself, using credentials a central service issued to that name moments ago. A compromised agent cannot reach past its own hostname. Below that, the entire grant of operating system privilege is three sudoers lines — create a cPanel session, list accounts, control the process supervisor. Everything else runs as the customer.

Shipping to a fleet you cannot push to

There is no deploy step that touches a hosting server. Publishing a release on GitHub builds the PHAR, ships it to a repository host, and tells the monitoring service a version number. The push side ends there.

Everything after that is pull. Each server runs an updater from cron every seven minutes. It sleeps a random one to ninety seconds first, because several hundred machines waking on the same cron minute is a denial of service you inflict on yourself. It takes a lock file that expires after ten minutes, compares the installed PHAR’s version against the monitor’s, downloads what changed, installs a WP-CLI shim, and restarts the consumers through the process supervisor.

The best idea in it is that the updater updates itself first. Before anything else, it downloads a fresh copy of itself, compares checksums, and if they differ it swaps and exits, so the new logic runs on the next cycle rather than half-executing old logic against new artifacts. Rolling back is deleting a directory on the repository host and re-announcing the previous version; the fleet downgrades itself within seven minutes without anyone connecting to anything. One line in that script is killall wget, to clear stragglers from an earlier run. It works, and it is a blunt instrument on a shared machine that assumes nothing else on the box uses wget.

The same release pipeline that builds the PHAR also gates what is allowed into it. A custom CI check fails any pull request where a parameter that looks sensitive isn’t marked with PHP’s #[\SensitiveParameter] attribute, which keeps its value out of stack traces and out of anything that serializes a stack trace. The project README instructs developers to name variables $password and $apikey rather than $pass and $key, not for readability but so the linter can find them. Hanging a security control off a naming convention costs nothing, and it held for two years.

Installation was stranger than deployment. We had no inventory of servers; WHMCS did. So nothing in TurboHub knew a new VPS existed, and what triggered the agent’s installation was a customer loading their WHMCS client area — a hook checked whether their server had an agent and, if not, asked TurboHub to install one. Removal ran the same way, on service termination and on a dedicated IP change caught by diffing the service record before and after a staff edit, so a reassigned box didn’t keep an orphan. Taking “WHMCS is the source of truth” literally has a bill, and this was it: the event that put our software on a machine was somebody visiting a page in another application. It behaved better than it sounds, and it sounds bad every time I describe it.

Running WP-CLI on someone else’s WordPress

Every WordPress operation the platform performed was a WP-CLI invocation, and the shape of that command is most of the security model. It runs under sudo -u as the customer’s own Unix user, never as root. Every argument is escaped. Plugins and themes are skipped unless the operation needs them. A shim is required into every run to force the WordPress locale to English, because output parsing that works on an English site fails silently on a German one. And the PHP binary is discovered per site, because a CloudLinux box and a standard one keep it in different places and the customer’s chosen version matters.

Then there is the blacklist. Customer plugins fatal on load. Not occasionally — constantly, across thousands of sites, in ways that have nothing to do with us and everything to do with a plugin that assumed some constant would be defined. A fatal during WP-CLI bootstrap breaks the command, which breaks the feature, which produces a support ticket about a button that doesn’t work.

So the agent learned to route around it. When a command fails, it parses the offending plugin’s slug out of stderr, refuses to blacklist any of the four plugins the platform itself depends on, appends the slug to a local file, posts to Slack so a human knows, and retries the same command with that plugin skipped. A guard remembers the last plugin it blacklisted; if the same one fails twice in a row it throws and reports “Prevented infinite loop” rather than trying again.

That is a hack and it should be named as one. We were writing to a file on a customer’s server based on parsing an error string whose format we did not control and nobody guaranteed — a WordPress or PHP release changing that message would have turned self-healing into silent breakage. And the infinite-loop guard is not a safety feature; it is a confession that we had built something capable of looping forever on a production box, and knew it. What justifies it is the alternative. The correct fix is auditing every plugin on the platform, which is not a task that finishes. This shipped in an afternoon and turned an unfixable class of ticket into a Slack message.

Single sign-on ran on the same machinery. cPanel was a one-shot session URL minted through the control panel’s own API; WordPress was a one-time login token from a plugin A2 maintained, where the call reads the plugin’s version out of the returned URL and quietly updates it when it’s stale. It is the one command that skips only the blacklisted plugins rather than all of them, because the login plugin has to load in order to answer. A separate check looks for customer-installed login-hardening plugins that would break the flow, disabling the button rather than letting it fail: the difference between a feature that works and one that works when you’re lucky.

Staging on a panel with no staging API

Customers wanted staging sites. cPanel and WP Toolkit have all the pieces; what they don’t have is a staging API. So creating one is a twenty-step orchestration of individual control panel calls, and every step carries its own named failure constant with a customer-facing message and a flag saying whether retrying is worth anything.

The sequence: read the account’s limits; raise the MySQL database quota if it isn’t unlimited, because the clone needs one more database than the customer paid for; switch on the toolkit capability the clone requires and put it back afterwards; add the subdomain; add the parked domain; create the DNS record on a staging domain, through WHMCS, through its DNS provider; find or register the site’s toolkit instance; clear caches; clone; trigger certificate issuance; verify with dig; then loop, waiting on SSL validation that answers whenever it answers.

None of that fits in ninety seconds, so it doesn’t try. The RPC handler special-cases staging creation: it acknowledges the message, launches the work detached from the connection, and never replies. Completion arrives minutes later as an entirely separate message on the web application’s own queue, or as a failure carrying the constant that identifies which of the twenty steps gave up. A cleanup routine unwinds partial state — a subdomain created before a clone failed is a subdomain that has to go away again.

The debugging story is the part I’d defend hardest. Each run narrates itself into a Slack thread, with emoji reactions used as status: the thread opens when work starts, every step appends to it, and the reactions change as it succeeds or fails. That sounds like a nicety and it isn’t. When you hand-build a compensating transaction across a vendor’s API because the vendor gave you no transaction, the record of what actually happened is the recovery tool. There is nothing else to look at.

This was the second implementation. The first one shipped, met production, and was replaced, and both versions were still in the tree at the end. Rewriting under load looks like that. You don’t get to delete the old one until the new one has been right for a while.

The database we said we wouldn’t build

Decision five in the spec was that the new application would not have a relational database of its own. The rationale was sound — “keeping two disparate databases in sync is a difficult challenge that is best avoided for as long as possible.”

TurboHub shipped with a database.

It broke on a question the API could not answer: which of our customers is running a vulnerable plugin? WHMCS knows about services. It does not know what WordPress is installed underneath them, let alone which plugins, at which versions. Nothing did. We had to go find out.

Discovery ran from the agent. A scanner walked every server, used WP-CLI to locate every WordPress install under the home directories, validated each cPanel username against the control panel, and forked a subprocess per site so one pathological install couldn’t take the scan down with it. It held a PID lock so two scans couldn’t overlap, and kept a per-day log of what it had already covered, so an interrupted run resumed instead of restarting. Each site’s data went back as its own message.

On the web side, the discovered cPanel username was resolved back to a WHMCS service — the one call in the system whose job is turning something we found into something WHMCS already knew — and then the site, its plugins, and its users were upserted. Site IDs are deterministic: a hash of the service and the path, used as a string primary key rather than an autoincrement. That single choice is what makes the pipeline idempotent. A scan can run twice, or partially, or after a site moves, and still converge.

With the read model in place, vulnerability tracking becomes possible. Patchstack’s database imported hourly. Matching a plugin against it is fiddlier than it sounds — patched ranges take priority over a single fixed version, leading vs get stripped, free and premium editions collide by name. Negative results cache for a day and positive results cache forever, which is not a performance trick but a statement of fact: a version that was vulnerable does not stop being vulnerable. From there, plugins with auto-update enabled were updated and everyone else got an email through WHMCS’s own template system, flagged so nobody was told twice.

A relational database against a spec that said none. Breaking that decision was right, and the discipline that mattered wasn’t holding the line. It was breaking it late enough, and narrowly enough, that what we built stayed a read model. Nothing in that database was authoritative. Every row could be dropped and rediscovered. A cache and a second source of truth differ in exactly that, and which one you end up with is decided by what you allow yourself to write there.

All hands, or one query

Sometime in 2024 the hourly Patchstack import came back carrying a critical remote-code-execution vulnerability in a plugin a lot of people had installed. Patched version already out. The platform’s inventory at that point covered roughly 385,000 WordPress installs, and the shape of that morning without an inventory is easy to picture, because it’s what everyone in shared hosting had always done: every engineer available, pulled off whatever they were doing, walking the fleet by hand to find out who was running a vulnerable version — and then, having found them, updating them. Days of it, probably weeks, with the error rate you’d expect from work that tedious.

Instead it was a query. The count that came back was in the thousands rather than the hundreds of thousands, which was the first useful thing to learn, and the second was that the affected sites did not all need the same thing done to them. One pass over the read model split them three ways. Sites with auto-updates enabled needed nothing from us at all — the path already in place would carry them by end of day, and they could be set aside rather than worked. Sites with auto-updates off needed us to do it. And a third group was running a version of WordPress core too old to accept the patched plugin, so the fix could not be applied to them at all; those got flagged out of the automated pass and handled by hand on a longer timeline. Sorting the population before touching any of it is what made the rest cheap.

The remediation itself used machinery that already existed. Updates went out as wordpress.plugins.update calls over the same agent queue everything else used — small batches first, checked, then wider, then the whole affected set. Three days, start to finish, for work that would have consumed the engineering staff for weeks. No customer complaints.

Nothing in the spec said incident response. The read model was built to answer a narrow question — which of our customers is running a vulnerable plugin — so that we could send them an email about it. What the emergency revealed is that the same table answers a different question just as well: not who should be warned, but who must be fixed, and in what order, and which of them can’t be. We hadn’t designed for that. We’d built an inventory for one purpose and found out, under pressure, that an inventory is simply what you need when you are under pressure. That’s luck. The part that wasn’t luck is that the data was still accurate on the morning we needed it, which is what a read model you can re-derive at any time buys you and a stale second source of truth doesn’t.

Automatic Optimization

The largest thing we built was a system that made a customer’s WordPress site faster with no human involved. It carried its own queue supervisor with a fifteen-minute timeout, because the default sixty seconds was not remotely enough.

Deciding what’s wrong. Analysis fans out in parallel across a dozen checks — PHP and WordPress version floors, required plugins, whether a CDN is in front, disk space, conflicting image optimization plugins, conflicting caching plugins, object cache, PHP opcode cache, page builder detection, and whether the REST API is reachable at all. The conflict lists are long and specific: a dozen image plugins, seventeen caching plugins, fifteen page-builder slugs. “Is another cache in play” has no general answer, only an enumerated one, and keeping that enumeration honest was ongoing work rather than a one-time cost.

Turning the dial. Optimization walks LiteSpeed configuration presets in order — Original, Essentials, Basic, Advanced, Aggressive, Extreme — snapshotting what it’s about to change before it changes anything, so there is always something to go back to. It climbs until a preset stops being an improvement or stops being safe. Sites built with a page builder stop at Basic by rule, because page builders and the aggressive presets disagree in ways not worth rediscovering one customer at a time.

The check that made it safe. Here is the part that mattered. “Did this optimization work” cannot be answered with a performance score, because every one of these presets makes the page faster and some of them make it wrong — a deferred script that broke a slider, an inlined stylesheet that dropped a font. So each preset is measured and photographed: the platform benchmarks the page with Google PageSpeed and takes a screenshot, then compares that screenshot against the baseline using a perceptual hash. If the visual difference exceeds a threshold, the preset is rejected no matter how good its score was.

That test is the feature. Speed is easy to measure and easy to over-optimize, and the only honest definition of success is that the page still looks like the page. Writing the acceptance criterion as visual regression rather than as a number is what made it safe to run for customers who had not asked anyone to touch their site.

Undoing it. Rollback is first-class rather than an afterthought. Individual actions have their own rollback paths, snapshots restore, and a scheduled job restarts optimizations that stalled midway. Every action taken is written to a log — and, in a nice inversion, those logs surface inside WHMCS’s admin reports, so staff read TurboHub’s data through the application TurboHub was never permitted to read from.

The unflattering footnote is that the optimization tables were dropped and recreated three times inside six weeks. We did not understand the shape of the data until the pipeline had run against real sites, and pretending otherwise would only have meant guessing for longer. Rebuilding a schema three times while shipping is not a failure mode. Refusing to is.

What we’d do differently

The front end is the biggest regret. The plan was to redesign the WHMCS client theme and bring TurboHub along with it. What happened instead was that TurboHub inherited a Bootstrap theme from the application it was supposed to modernize, and then began migrating to Inertia and Vue one page at a time. At the end there were seventy-five Blade templates and forty-nine Vue components, and which one you got depended on when that page had last been touched. Half-finished migrations are not free. They double the number of patterns a developer has to hold, and they make “how do I add a page” a question with two answers.

Thirty-seven test files, and every one a unit test. For an application whose entire behavior is an HTTP boundary with a vendor — forward these cookies, interpret this permission string, handle this response code — testing services in isolation covers the less interesting half. The middleware that decided who you were had no test that exercised a request. We knew. It never won a prioritization argument, and losing that argument enough times is how a gap becomes permanent.

What held up was the prohibition. Never touching the WHMCS database meant every dependency on it was a call site we could enumerate, and enumerable dependencies are what make a system replaceable. Wrapping third-party clients did the same for the vendors. Making the queue name the server’s identity meant addressing and authorization could not drift apart. Deterministic site IDs meant a discovery pipeline that could be re-run at any time without fear. And keeping the broker that thin meant WHMCS never learned to depend on us.

The exit never happened. A2 Hosting was acquired by World Host Group in January 2025, and the last commits on both repositories land that same month. The WHMCS Exit Plan™ stayed a section in a document.

Nobody plans for that ending, and it is easy to draw the wrong lesson from it. The architecture worked: after two years, the list of things standing between A2 and leaving WHMCS was the same list the spec had written down, and it had not grown. But what the discipline actually bought was not the exit. It bought two years of shipping features into a control panel that could not otherwise be changed, for a company that needed them, which was the whole point of the iterative approach and the reason the option was worth holding even unexercised. Optionality is only wasted if you’d have made the same decisions without it. We wouldn’t have.