Enterprise organizations, defined as those with 5,000+ employees, are typically “lower performers” on measures of software delivery and operational performance. Within the approximately 3,500 US-based enterprise organizations (between commercial companies and government), there are large variations in scale by firm and sector.
For instance, retailer Walmart has in excess of one million employees, with Amazon close behind at approximately 800,000. The top five retailers average 797,000 employees (the most per sector). The US Military top five-branch and service components average 349,000. By comparison, SpaceX has only 8,000 employees, just 6% of the aerospace and defense top five sector average of 141,000.
Defining DevSecOps
Before exploring the challenges within DevSecOps, we need to start with the definition used by the DoD:
“DevOps is a software engineering culture and practice that aims at unifying software development (Dev) and software operation (Ops). The main characteristic of the DevOps movement is to strongly advocate automation and monitoring at all steps of software construction, from integration, testing, releasing to deployment, and infrastructure management. DevOps aims at shorter development cycles, increased deployment frequency, and more dependable releases in close alignment with business objectives.
“DevSecOps adds baked-in security to the continuous integration/continuous delivery (CI/CD) process, including static/dynamic code analysis, fuzzing, and bill of material assessment. More importantly, continuous monitoring is critical, with behavior detection and zero trust enforcement.”
Driving DevSecOps Change at Scale: Air Force and DoD
Recently, I interviewed Chief Software Officer of the United States Air Force, Nicolas Chaillan. He also currently serves as the co-lead for DoD’s Enterprise DevSecOps Initiative efforts alongside the DoD CIO and A&S.
As the first-ever Chief Software Officer, Chaillan is responsible for enabling Air Force programs that are in transition to agile and DevSecOps to establish force-wide capabilities and best practices. These capabilities and practices include continuous Authority to Operate (cATO) and streamlined technology adoption across the DoD. He also brought with him many years experience in the commercial space, ranging from cloud computing, cybersecurity, big data, multi-touch to mobile and VR. Chaillan is an accomplished entrepreneur with successful ventures — a background that allows him to apply a powerfully unique lens and approach on the mission and future state of the Air Force.
In my interview with Chaillan, we discussed three major challenges and five DevSecOps principles he’s focused on right now. He also shared more about the metrics around these efforts, and how he’s measuring success for the organization.
Challenge #1: Organizational Silos
The first challenge Chaillan shared is the existence of countless organizational silos and how the sheer size of DoD further aggravates this common organizational challenge. It’s a large organization with a complex mission (e.g., space, cyber weapons) and a complex organizational structure, making it tough to get agreement and alignment. DevSecOps’ success requires a culture of continuous learning, as “A college degree is obsolete in two years,” and “Software principles that used to be good for ten years are now only good for two to three years at best.” Embracing continuous change is required given software’s accelerating, but challenging in a siloed environment.
How Addressed:
- Conduct an “audit” to understand and identify where silos exist — and gain awareness of the stakeholders and critical parties within each.
- Shift personnel to embed all necessary talent into smaller teams of no more than seven to twelve people (2-pizza rule).
- Provide briefings tech and its value.
- Get commitment to continuous learning and understand the need for continuous learning.
Challenge #2: The “Bubble”
The second challenge identified is the need to protect classified information, which when paired with the challenge of an organizational structure with silos, results in a “bubble” of government software architecture. The “bubble” refers to a closed system, and is specifically a concern around weapons.
“This puts the country at risk. We must embrace open architecture, open source tools and the brightest and latest principles of commercial best practices.”
Related to this, the DoD historically gave considerable control to Defense Industrial Base (DIB) vendor contractors. Over the many years of efforts teaming with vendor organizations, the DoD culture began to shift to believe that the government should not even be involved in the architecture decisions or development on a substantial level.
For example, the mix might have been 90% contractors or DIB, and 10% government. The purchasing approach was often pay-for-hire (time), instead of buying IP or pay-for-use.
This historical approach no longer serves the modern-day missions and goals of government organizations, so the team is working diligently alongside industry to change that ratio to something more balanced, e.g., 70% contractors, and 30% government.
How Addressed:
- Shift mindset to “our mission is special but our software should not be.”
- Require DoD and big prime contractors to embrace open architecture, open source tools and commercial best practices.
- Migrate mix (which will vary considerably) to approximately 30% government, 70% contractors.
- Focus on training for government cloud native architects and instilling the mindset of test-driven development (TDD) and domain driven design (DDD). Historically, the architect role has often been completely missing.
- Move to microservice architecture and ensure DoD primes move away from monolithic application architectures. A small, containerized workload will enable the DIB to productize their capabilities and make more revenue.
- Adopt a pay-per-use/consumption models. “This is better for the contractors as they go from being a service company to a product company with recurring revenue. You can’t get huge multiples of revenue for the valuation of a service company because they are just pure service, whereas you can see as much as 20X to 50X revenue with a product-based company.”
Challenge #3: Timeliness is critical to keep up with the pace of relevance
Given the speed of technology change, rapid integration of new software technologies is vital. Historically, onboarding a new contractor could take so long that the original purpose for using them became becomes obsolete.
“The effort will fail if the DIB is not brought along with us. Initial engagements at Kessel Run were mostly airmen… We need to make it much easier for companies who have never done business with us to bring us their innovation.”
How Addressed:
1. Streamline accreditation to two to three weeks. Streamline acquisition. Currently there are 25-30 companies with products accredited in the initial six months of the program. This saved on average 12 to 18 month of planned time for every five years, per program, and the total time saved in one year was 100 years.
2. Build a marketplace that spreads the word that these tools exist, and drive awareness.
3. Have early, tangible wins on tough challenges to prove success. “People need to see success. We placed containers on the F-16 legacy hardware. We proved that it can be done in 45 days. If it can be done on jets, it can be done anywhere.”
Five DevSecOps Principles to Focus on
According to Chaillan, there are five DevSecOps principles for organizations to focus on that apply across organizations. These principles are broad, and apply in the commercial sector as well as the governmental space.
- Centralize DevSecOps and adopt high-performing practices or become obsolete. “Look at Tesla and SpaceX. It’s all a DevSecOps platform. SpaceX can update the rocket’s software a day before the launch versus freezing the code 60 days before. Tesla can send over the air updates every few cars to their existing car fleet. Any organization that’s touching software and doesn’t have a centralized DevSecOps team will become obsolete in less than three years. That’s one of the first things I look at when looking at a company.”
- Use chaos engineering, test and see if it can be broken. “Is it modular and flexible?”
- Avoid obfuscation, and embrace open architecture. “Obfuscation gives a false sense of security. Fear of bad actors is used to obfuscate. Many healthcare and financial organizations are behind in this sense and put data at risk. Often done under the false pretense of complexity and regulation.”
- Embrace a culture of continuous learning.
- Measure progress and learn from the leading organizations. Chaillan mentioned that some of the enterprise leaders in DevSecOps are SpaceX, Tesla, Netflix, and JP Morgan. Many other enterprise organizations have some limited adoption. Those that choose not to adopt any of the best practices can expect to become obsolete, given the vast cost and time-to-market differences. A set of metrics for DORA are provided. These include:
- Deployment Frequency
- Mean Lead Time for Changes
- Time to Recover
- Time to Production
- Change Failure Rate
Conclusion: Embrace DevSecOps or Become Obsolete
Differences between elite and lagging performers are dramatic. To date, considerable progress has been made, including saving 100 years of planned programming time in the first year, with dozens of teams adopting the approach, including the large programs like the AEGIS, F-35, Ground Based Strategic Deterrent (GBSD) and the F-16. The programs are viewed as customers, and adoption is not mandated.
This example of DevSecOps at enterprise-scale within the Air Force and DoD highlights how crucial it is for enterprise organizations to become adept at DevSecOps. Without this technology evolution, many organizations are at risk of becoming obsolete.
