Helix

Helix was the billing and hosting platform behind World Wide Web Hosting, Pressed, and Envato Hosted. It started in 2011 as a ground-up replacement for a legacy billing system, and over the next eight years it followed the business through two acquisitions and a complete change of product, from traditional shared cPanel hosting to white-labeled managed WordPress. It handled signups, orders, invoicing, payments, refunds, partner payouts, domain registration, SSL issuance, DNS, support, and the full provisioning lifecycle of every site on the platform, across dozens of brands that each looked like an independent hosting company.

The first commit lands on September 6, 2011 and reads helix: Rails 3.1. The last lands on June 26, 2019, has no diff at all, and reads Shut down today at 2:57am. Between those two commits: about twenty people through the door, and three companies that each thought of it as theirs. This is a look at how it was put together, including the parts we got wrong.

The shape of the problem

Helix was really two systems welded together, and most of what was hard about it came from that seam.

On one side was a billing system, which has to be exactly right. Money is not eventually consistent. An invoice that double-charges, a refund that silently fails, a partner payout that’s off by a rounding error — these are not bugs you fix in the next release, they’re bugs you apologize for.

On the other side was a control plane driving physical infrastructure, which is never right for long. Provisioning a site meant creating a Unix user, a database, a virtual host, a DNS zone, a TLS certificate, a PHP process pool, and a WordPress installation, across several services that each fail independently, on their own timelines, for their own reasons. Nothing about that is transactional.

Getting those two halves to share a codebase without contaminating each other — keeping billing deterministic while provisioning stayed asynchronous and retry-safe — is the engineering story of the platform. Everything below is downstream of it.

Four applications, one engine

Helix was four Rails applications sharing a single engine.

HelixCore was the engine, and it held nearly everything: the models, the background workers, the payment gateways, the registrar and certificate integrations, and every migration. The three other applications depended on it through a path-based gem reference and were, by comparison, thin — mostly controllers, views, and per-brand overrides.

Helix, the admin application, was the staff surface and by far the largest: customer management, order review, fraud triage, provisioning diagnostics, financial reporting, brand configuration, and refunds.

Build (originally Backstage) was the customer-facing application, deployed per brand on the partner’s own domain, where customers ordered hosting, managed their sites and domains, updated billing, and opened tickets.

PartnerPanel, the smallest of the three, was the reseller surface: partners managed their own customers, pulled revenue reports, and configured their brand.

Behind all of them: one PostgreSQL database, one Redis, and a Rails version that started at 3.1 and finished at 5.1.6.

Now the part we got wrong. The engine relationship was backwards. We built HelixCore as an engine that each application included, when it should have been a single core application with each surface — admin, customer, partner — as an engine mounted inside it. By the time that was obvious, four applications and several years of history were sitting on top of the decision, and unwinding it was never the most valuable thing we could do that quarter.

The cost was paid on every deploy. Any change to shared code meant deploying all of them. Correcting a typo in a translation string was three deploys. For a while it was four, because the order form lived in yet another application consuming the same engine; folding that back into Build was the one structural correction we managed to land, and it measurably improved life. We eventually taught CI to skip an application’s test suite when nothing in it had changed, which helped — but that is a workaround for a shape problem, not a fix.

White-label as a data model

The thing that survived all three companies unchanged was the brand system, and it survived because it was a data model rather than a configuration file.

A Brand record owned its own domain, its own TLS certificate, its own payment gateway credentials, its own registrar credentials, its own nameservers, its own support configuration, email templates, product catalog, pricing, and theming. Brands were resolved per request by matching the incoming host against a domain suffix, so a single deployment served every partner and figured out which one you were from the URL.

Credentials were the interesting part. Each brand carried its own encrypted gateway logins, and a separate record mapped currency to merchant account, so the same Rails code charging the same card type would route through a completely different merchant depending on which partner’s storefront the customer had walked in through. Field-level encryption was handled inside PostgreSQL using pgcrypto rather than in application code, which also meant that bulk reconciliation jobs could decrypt in SQL and avoid pulling tens of thousands of rows through ActiveRecord one at a time.

Creating a brand kicked off real work: DNS records, a Let’s Encrypt certificate request, mail configuration, and a full default product catalog — hosting plans, staging, DNS, the common TLDs, SSL products — generated automatically so a new partner started with a working store instead of an empty one.

Theming went further than a stylesheet. Each brand got its own compiled CSS bundle, kept out of the main application bundle and precompiled separately, plus its own logo served from object storage. But brands could also override any view. A before_action prepended a brand-specific view path on every request, so dropping a file into a brand’s directory replaced that partial for that partner only, with no conditionals in the shared template. By the end more than two dozen brands were carrying their own view overrides and their own stylesheets. It is a blunt instrument and it can rot — an override silently keeps rendering the old markup after the shared version moves on — but it let a partner ask for a genuinely different checkout flow without anyone forking the application.

Products as classes

Every sellable thing resolved to a driver class under a ProductTypes:: namespace, and that hierarchy is where product-specific behavior lived while billing stayed generic. Domains had OpenSRS and Enom drivers with per-TLD subclasses for registries with special contact requirements. Certificates had their own. WordPress hosting had a driver for the billed parent plan and separate drivers for child sites, staging environments, and DNS-only services. Addons covered plugins, plugin packs, themes, must-use plugins, and storage upgrades.

A Service did not know how to provision itself. It asked the product it was sold from which driver class handled it, instantiated that driver around itself, and delegated. The bridge between the two layers was a single table mapping state machine events to driver methods — activate to provision, terminate to deprovision, suspend to suspend, change_domain to change_domain — so adding a new kind of product meant writing a driver that answered those calls, not touching the billing code.

Eleven models carried explicit state machines rather than relying on callbacks and boolean columns: Service, Order, Invoice, Account, Backup, DomainService, and others. Service moved through pending, activating, activated, suspended, terminated, with side transitions for holds, cancellations, and domain changes. Order moved from open to submitted to invoiced to paid to complete, with a branch to a manual-review state when a fraud score crossed a threshold or a product demanded human eyes. Because checkout was a chain of state transitions rather than one long method, an order that failed partway through simply stopped at its last good state and could be resumed instead of unwound.

One exception was deliberate, and it shows where the two halves of the system disagreed: charging an invoice was implemented outside the state machine on purpose. If the transition failed, the state machine would roll back the whole transaction, including the Payment record documenting that we had, in fact, just taken the customer’s money. Losing that record is far worse than an invoice stuck in the wrong state.

From a button click to a running site

Rails never touched a hosting server directly. It handed work off, and it did so four different ways depending on what the work was — which sounds like a mess, and partly was, but each shape earned its place.

A synchronous REST API for creating the flat resources a site needs to exist at all — the Unix user, the database, the certificate, the site record. Fast, immediately consistent, retried on gateway timeouts. Internally this layer was called Cloudburst, and it doubled as a reconciliation target: a bulk push could re-sync every service in a region if the two sides ever drifted.

A message queue with an HTTP callback for the slow lifecycle operations. Rails put a command on an SQS queue; a small service on the other side named Emeril consumed it and executed the corresponding shell script; when the script finished, a second queued job POSTed the outcome back to a callback endpoint in Rails, which advanced the service’s state machine. This carried provisioning, deprovisioning, suspension, resumption, domain changes, SSL refreshes, and WordPress version upgrades: anything that might take minutes.

Fire-and-forget queue dispatch for work with no user waiting on it: backups, restores, database drops, and exporting a live site into a reusable archive.

A pull-based configuration API for the server fleet itself. Rather than pushing configuration out, Helix exposed an endpoint that rendered the current state of every service in a region as structured configuration data. SaltStack minions on the actual servers polled it and reconfigured themselves. The Rails database was the source of truth for infrastructure, without Rails ever needing to know how many servers existed.

The detail we’d defend hardest is the contract at the bottom of the second shape. Every lifecycle script read a single JSON document from standard input. That is the entire interface between a Rails monolith and a fleet of Linux servers. It meant an engineer debugging a failed provision could pipe a payload into the script by hand and watch it run. It meant the scripts had a real test suite: RSpec, driving actual LDAP and MySQL inside a container, asserting against the resulting filesystem and database. And it meant the command vocabulary was just the list of filenames in a directory: provision, deprovision, suspend, resume, change_domain, change_password, refresh_ssl, acme_challenge, flush_redis, impersonate.

Provisioning a WordPress site fanned out rather than running in sequence. Separate components handled the WordPress install, the DNS zone, the CDN distribution, the certificate, and an HTTP health check, each dispatched independently and each retried on its own. Completion was converged by polling: the service carried a timestamp column per component, and the state machine refused to move from activating to activated until the required ones were all filled in. Steps that depended on earlier ones scheduled themselves only once their prerequisites had reported done. It is not elegant, but it is honest about the failure model — any one component can fail and be retried without redoing the others.

What actually served the sites

The hosting platform underneath was assembled rather than bought, because in 2015 most of it could not be bought.

Varnish sat at the edge. Cache purges went through a small daemon that authenticated callers by checking the peer credentials on the Unix socket — a neat trick, since it means authorization is enforced by the kernel rather than by a shared secret that can leak. nginx terminated TLS, HAProxy handled load balancing, and Apache served sites in mass virtual host mode: instead of a virtual host file per site, generated flat maps translated domain to document root and domain to PHP socket, regenerated whenever a site was provisioned or changed domains.

PHP ran as one FPM master per site, which is the isolation model you want and not the one PHP makes easy. Doing that at scale needed its own tool — a small program that rendered a systemd unit and a pool configuration per site from a template, injecting database credentials and WordPress salts as process environment rather than writing them into wp-config.php on disk. Every site also got its own Redis instance for object caching and sessions, addressed by its own socket.

Databases used a read-replica router, patched and loaded through PHP’s auto_prepend_file rather than dropped into the WordPress tree — deliberately, so customers browsing their own files never saw platform plumbing sitting in wp-content and never had the chance to delete it. The same mechanism delivered the object cache integration and a numbered pipeline of environment setup that ran before WordPress did.

Site files lived on a network filesystem, which is what made the whole thing scale and also the source of most of its pain. Sites were sharded across pool clusters, with tooling to migrate a site between clusters or between storage backends by rewriting its configuration and repointing an internal DNS alias. Monitoring and log shipping ran through agents on every host; backups ran to per-region object storage.

Bundles

Bundles were the most interesting thing we built, and the clearest example of the two halves of the platform having to cooperate.

The problem is easy to state. Someone buys hosting bundled with a WordPress theme. What they want is the site from the demo. What WordPress gives you is an empty install and a zip file, and a theme’s demo content typically only imports correctly through a sequence of admin screens that assume a human is clicking them. Doing that by hand, per customer, does not scale. Doing it at provision time, live, on a server, is slow and fails in ways you cannot debug afterwards.

So we moved the work off the provisioning path entirely and turned a configured WordPress site into a build artifact.

Building one. A theme purchased from a marketplace arrives as a messy archive — the theme, child themes, required plugins, demo content, documentation, licenses, sample files. A parser sorted that into the parts worth installing and discarded the rest by pattern. A per-theme JSON manifest, checked into git, declared the build parameters and one entry per design variant, since a single theme might ship twenty different demo looks. A CI job then provisioned a real WordPress install inside a container, installed the theme, activated the plugins, imported the demo content for that variant, took screenshots, and packaged the result. Rails tracked each build as a state machine — pending, running, finished, failed — and the build record’s columns were handed to the CI job as parameters by naming convention, so flags like “seed posts”, “seed the menu”, “skip screenshots”, or “verify PHP 7 compatibility” were data rather than code.

What a bundle was. A versioned archive in object storage containing exactly three things: the site’s wp-content minus the platform’s own must-use plugins, a database dump with the users tables’ structure but none of their rows, and a small JSON file recording which values in that dump should be treated as placeholders. That third file is the whole idea. A bundle is a real site with its identity extracted and written down.

Installing one. At provision time, the freshly-created site’s wp-content was replaced with the bundle’s, the dump was imported, and then the templating was run in reverse. wp search-replace passes swapped the build machine’s document root, the build URL, the build’s admin email address, and a literal theme-name token for the new site’s real values. A final pass rewrote absolute URLs to relative ones, so the site would survive being moved to a different domain later — which, for a customer who orders hosting before deciding on a domain name, is the normal case rather than the exception. Then the real administrator was created, every user carried over from the build was deleted with their content reassigned to that new account, and wp core update-db ran, because bundles are built once and WordPress keeps moving.

The test that mattered. The acceptance test for all of this does not check that the site works. It checks that the provisioned database contains none of the placeholder strings — not the build document root, not the build URL, not the placeholder email address, not the theme token. Templating that half-works is worse than templating that doesn’t, because a leaked placeholder is invisible until a customer finds it in their own site’s footer. Writing the test as leak detection rather than as a happy path is the single decision that made the feature safe to iterate on.

Running it backwards. The same machinery worked in the other direction: take a live, customized site and snapshot it back into a bundle, stripping its real users on the way out. That inversion is what turned an internal tool into a product. It meant a theme author could build their demo site by hand, on real hosting, the way they wanted it — and then publish it as something customers could buy and receive in minutes. The feature moved from staff-only bundling, to CI-driven builds, to author self-service, and each step gave more control to the people who actually knew what the site was supposed to look like.

The unglamorous half was drift. Themes update, plugins update, WordPress updates, and a bundle built eight months ago starts failing in new ways. Helix tracked outdated bundles as first-class records, retriggered builds automatically, and filed tickets when they broke. An artifact pipeline is only as good as its answer to staleness, and staleness is relentless.

Making WordPress behave

Platform behavior was delivered through must-use plugins loaded from a directory outside the customer’s own tree, so customers could not disable or delete them and they did not appear in their plugin list. That framework carried impersonation, cache flushing that reached both Varnish and PHP’s opcode cache, suppression of core update prompts (the platform owned WordPress versions), and a growing collection of patches for specific plugins and themes that misbehaved in a managed environment.

The impersonation pattern outlived the platform by years. Staff and customers could jump straight into wp-admin from the control panel with no WordPress password involved: Rails minted a short-lived single-use token, the request carried it to the site, and a must-use plugin validated it and established a session for the right user. Every use was written to an audit log naming who impersonated whom. We built the same pattern again years later, at a different company, because it is still the right answer.

Then there was the upgrade problem. WordPress upgrades a plugin by copying it into place file by file. On a local disk that is fine. On a network filesystem, with a plugin containing a few thousand files, it took over two minutes, long enough to hit PHP’s execution timeout and leave a half-installed plugin behind.

The fix was to replace the filesystem class underneath WordPress’s upgrader. Instead of deleting the old version, move it aside into a trash directory. Instead of copying the new version in file by file, symlink it into place — one operation instead of thousands. Report success to the user, then materialize the symlink into real files and empty the trash in a background job after the response has gone out. The directories that made this cheap were pre-created per site at provision time, on local disk rather than the network share. Perceived upgrade time went from over two minutes to about thirty seconds.

That is a hack, and it was written down as one at the time. It reaches into WordPress internals that carry no compatibility guarantee, and every core release was a chance for it to break. But it solved a problem that was actively costing customers their afternoons, it shipped in weeks rather than quarters, and it held in production for years while the underlying storage architecture was replaced properly. Knowing which problems deserve a correct solution and which deserve a fast one is most of the job.

Money

The billing half of Helix was less photogenic and had a far lower tolerance for error.

Amounts were a type, not a float. Every monetary value carried its currency, which matters enormously when a single deployment bills in several of them through different merchant accounts. Orders, invoices, payments, refunds, voids, credits, coupons, promotions, VAT, exchange rates, and ledgers were all modeled explicitly. Invoices could be paid from account credit without touching a gateway. Charging respected account state, so an account that was frozen or awaiting review got skipped with a logged reason rather than quietly charged anyway.

Card processing ran through two gateways. Migrating from Vantiv to Braintree meant running both simultaneously across every brand, selecting per brand based on which credentials were configured, and moving partners over without interrupting anyone’s active billing. We also maintained a fork of the payment library we used, to add refunding against a stored card token on the older gateway — the upstream support did not cover it, and our billing model required it. Carrying a patched dependency is a real cost, and we chose it knowingly over redesigning refunds around a gap in a library.

Because the platform was white-label, partner economics were part of the domain model rather than a spreadsheet. Payout records, per-service payout calculations, and a ledger computed what each partner was owed, with pricing carrying both the partner’s cost and the payout rules above and below it. A partner reselling your platform is entitled to a number they can audit, and that obligation belongs in the schema.

Around all of it: domain registration through two registrars, certificate issuance through commercial authorities, and automated Let’s Encrypt issuance and renewal from 2016, early enough that free automated SSL was still a competitive differentiator rather than table stakes.

What we’d do differently

The engine inversion is the big one, and it is a lesson about timing more than about architecture. We understood the mistake within a couple of years and never fixed it, because at every individual decision point there was something more valuable to do. The lesson is not “choose the right structure” — it is that structural mistakes get more expensive on a schedule, and the window to fix them closes quietly.

Four applications on one database compounded it. It meant deploys were coupled, schema changes were fleet-wide events, and no surface could evolve its storage independently. If we were starting again the boundaries would be drawn around data ownership first and applications second.

We also built a great deal of infrastructure that is now commodity. Per-site process isolation, certificate automation, a container build pipeline, edge caching with programmatic purge — in 2015 these were things you assembled from parts. Today most of them are a managed service and a configuration file. Building them was the right call then; the skill that transferred was not the code, it was knowing exactly which failure modes those services are papering over.

What held up: the JSON-on-stdin contract between the control plane and the fleet, which stayed debuggable and testable for its entire life. Per-site process and cache isolation, which meant one customer’s runaway site was one customer’s problem. Explicit state machines over implicit callbacks, which made partial failure a state you could look at rather than a mystery. Treating provisioning as fan-out with per-component retry instead of a transaction that pretends infrastructure is atomic. And bundles as versioned artifacts with a test written to catch leaks rather than to confirm success.

Helix served its last request on June 26, 2019. The final commit has no diff, just a signed marker saying the platform had been shut down at 2:57 that morning. We are building managed WordPress infrastructure again now, on serverless compute and managed services that did not exist when this started. The problems have not changed nearly as much as the tools have.