Page 48 - GCN, June/July 2018

Viewpoint
deadlines and no one will reward them for going out of their way to pay off this still invisible debt.
This process is
a pattern that has persisted for decades. It causes our government to accumulate a massive amount of technical debt. Millions of lines of completely undocumented, copy- pasted code written in languages that no one knows any more are running on systems whose creators are now dead. Some systems provide life-or-death services for people,
so the instability
that comes from this technical debt has real consequences.
The engineers building this software are set up to fail by contracts that reward checking boxes over delivering working and maintainable software. They also make any effort to pay off technical debt effectively charity. This is not a great situation for anyone.
A TWO-PART PROBLEM
U.S. Digital Service teams have built, debugged, read
and rewritten software. Rather than asking agencies for management-level reports, we sit down with the engineers to read and run the code ourselves. We can engage with them on a technical level and hear their deepest concerns — after reassuring them that we are there to fix the problem rather than find people to blame.
Often, when we ask for very specific technical information — such as “show us your deployment process” or “how do
we run your automated tests?” — people respond with a mixture of surprise, fear or even excitement. Many have never been asked those questions before. Some are afraid that the comfortable status quo will be disrupted, while others
are excited that someone in government finally cares about the issues they’ve been quietly keeping at bay (and thereby averting disaster) for many years.
Ultimately, the problem has two parts: hiring qualified people and incentivizing good infrastructure.
USDS helps the government hire skilled software engineers by developing “coding challenges” in which applicants for a contract must write code as part of the application. We can then evaluate the code based on real, objective results instead of how good the proposal sounds. Developers can claim to use “best practices,” but
if their code does not include any tests (or the tests don’t work), the truth becomes obvious
INCENTIVES TO PAY OFF TECHNICAL DEBT
Once skilled software engineers are hired, they can be set up to succeed by rewarding measurable, real-world infrastructure outcomes, which might include security (measured by penetration testing and bug bounties), robustness (measured by load tests and simulated failures) and maintainability (measured by how long it takes to onboard a new person and how many bugs are found late in the development process).
Understanding technical debt and knowing some of the strategies to pay it down are crucial to building robust, secure and maintainable software. The government provides critical services that affect people’s lives in dramatic ways, and every hack or shortcut puts those people at risk, now and for years to come.
Hiring skilled teams and incentivizing them to fix technical debt would go a long way toward improving software in the federal government. •
Shaun Verch is a digital services expert in site reliability engineering at the U.S. Digital Service. USDS is a nonpartisan startup at the White House that is building a better government with design and technology. This essay first appeared online in GCN’s sister publication FCW.com.
Millions of lines of completely undocumented, copy-pasted code written in languages that no one knows any more are running on systems whose creators are now dead.
48 GCN JUNE/JULY 2018 • GCN.COM