## Core Insight When automation is maintained separately from the system it manages, it suffers bit rot and creates perverse organizational incentives. The most functional tools are written by those who use them. Separating automation ownership from system ownership causes three predictable failures: the automation team has no incentive to reduce the service team's tech debt, the service team has no incentive to make their system easy to automate, and product managers unaffected by automation quality always prioritize features. ## The Cluster Turnup Lesson Google's cluster turnup evolved through four phases: 1. **SSH scripts** — quick initial win, became technical debt 2. **Prodtest** — Python unit tests validating real services, with dependency chains 3. **Idempotent fixes** — each test paired with a fix, safe to run every 15 minutes 4. **SOA-based Admin Servers** — RPC-based, ACL-driven, audit-trailed; service owners responsible for their own turnup RPCs The critical failure occurred between phases: when a separate "turnup team" ran automation on behalf of service owners, quality degraded because the automation maintainers weren't the system operators. ## The Principle Automation code, like unit test code, dies when the maintaining team isn't obsessive about keeping it in sync with the codebase it covers. The world changes around the code — new config options, new package names, new devices — and separated automation falls behind. ## Cross-Domain Applications - **DevOps**: "you build it, you run it" is the same principle — operational knowledge must stay with the development team - **Testing**: test suites maintained by a separate QA team suffer the same bit rot as separated automation - **Documentation**: docs maintained apart from code drift out of sync for identical reasons ## Source - [[Site Reliability Engineering - Chapter 7 - The Evolution of Automation at Google|SRE Ch 7: The Evolution of Automation at Google]] by Niall Murphy ## Related Concepts - [[Automation Value Hierarchy in SRE]] - [[Autonomous Systems vs Automated Systems]] - [[SRE 50 Percent Engineering Rule]] - [[Cognitive Load Cost of You Build It You Run It]] — Counterpoint: full-ownership models reduce coordination overhead but transfer real cognitive cost onto the owner; platform engineering exists to absorb some of it back - [[Incentive Mapping Framework]] — The general mechanism this note's failure mode instantiates: institutional and individual incentives diverge whenever ownership and consequence are separated - [[Hammurabi's Builder Law — Skin in the Game Origin]] — The ancient precedent: a builder who bears the consequences of their own work naturally maintains it well; separation is what breaks the alignment - [[Founder-Engineer Identity as Competitive Moat]] — Same mechanism at the org-design level: keeping authority and domain knowledge in the same hands eliminates the principal-agent problem