DevOpsDays LiveBlog – DevOps in Government (DoD)

DevOps in government pic of military torpedo boat

the good the bad and the ugly

By Peter Walsh

I want to share with you some of the process that we have implemented at the DOD. We don’t have the silver bullet but we have improved things.

The Ugly

DoD spends more than $32 billion per year on IT systems. In 2008 our software creation and deployment process we went from 38 documents and gates to well over 40. Yay, moving backwards! In the DoD releasing software requires multiple levels of tests and approval, all within distinct organizations that don’t much communicate.

The Bad

Normally you have a program office that pushes problems to solve/projects out to dev contractors, test contractors, test agencies, and ops teams. Each of these groups are quite separate. They all create their own setups, which are distinct, and which basically guarantee inconsistency from one group to another. They manage things by excel, and project. Costs are measured in piles of money, and time is always calendar time, everything is slow.

Another large challenge for DoD systems is this challenge of accreditation. Before anything can be used within the software creation process or delivered it must go through this process. Everything! Hardware, OS, middle ware, applications. This process is implemented by a separate agency. Each of the aforementioned teams goes through this process separately. This process limits flexibility and it restricts change. This is why the government lives on old technology.

Good (all is not lost)

There are 26 projects on a Whitehouse watchlist for improvement and experimentation, some are DoD projects. There is some recognition that we need to be more agile and perhaps Agile. There are a few new programs out there that are trying to enable everyone in the DoD along these lines. Some of the more prominent ones are listed here:

  • is for government open source projects. They look to create community tools and foster a bit of colaboration. The vice chairmen of the cheifs of staff supports this project.
  • Race is a government program for Cloud resources and accreditation assistance.
  • is another project that loks toward improving system provisioning, automated workflows, and test as a service.

Another project within the DoD that has done a lot to support the shorter lifecycles and higher quality delivery is called CONS3RT. We will discuss the approach it takes moving forward here

CONS3RT is a library of all the assets that you have from dev to operational deployment. Assets can be things like servers, databases, monitoring tools, etc… Assets are pulled together into what is called scenario which can be deployed into a cloud space in a repeatable fashion. This allows the different agencies and groups responsible for delivering software as described earlier to work on the same scenarios which eliminates some of the inconsistency between these groups. This also addresses the acredidation requirements. Once something has been accredited then it can be reused in this way between groups with out being reacredited. We realized that accreditation could not be just thrown away in building CONC3RT so it is quite well integrated into what we do. Peter and his group are not trying to tackle everything, but instead to start small and work within the boundaries of what can be changed.

This focus on accreditation is really key also because in the past developers did not want to go through the accreditation process becuase of the restrictions. They developed on their own hardware and then kicked things over the wall to move through the rest of the life cycle. This guaranteed nothing worked for the first dozen times it was kicked over. Now, this is less of a problem. Many other improvements have come out of the use of CONC3RT and this approach in general.

With this sytem reuse also improved. Instead of each organization learning from scratch how to deploy x or y spending piles of money doing it components already placed into CONS3RT can be used. This is something that virtualization and cloud in general can bring to any company.

Time is measured in minutes now a days instead of days and weeks. Things can be deployed with a couple mouse clicks instead of new hardware installation. More resources are also available to people now both in terms of the variety of software they can access as well as the CPU power they can harness through increased utilization. Tallent can be focused because people are not wasting time doing the job of provisioning, accreditation and other coordination activities.

How does software flow through the lifecycle now?

Stage – Developer: New code -> local IDE build -> Jenkins & Maven -> if build good, deploy to integration, if not, back to write code
Stage – Integration: Deployed to Integration -> Debug and test -> if tests pass send to CI if not, delete the artifact and go to dev
Stage – CI: Deploy to CI -> smoke/regression -> if pass -> if enough new -> tag -> send to QA env
Stage – QA: Deploy to QA -> Smoke/regression/manual/exploritory -> add new tests -> tests pass -> new tests push to past env -> tag new bits -> push to production

Most organizations in the government, and in industry, handle the writing of new code, building the code and pushing into production. The stuff in the middle, not so much, many orgs do not have much of a handle on how they are doing those functions. Our pipeline as mentioned above in conjunction with CONC3RT, maven, jenkens, and other tools is really helping us get a handle on it.

Lessons learned

  • Automation is huge.
  • Testing early and often. It is easier to fix the engine when it on the assembly line then when it is on the dealers lot. Simple concept. Virtualization and CI allow you to test often with minimal barrier to doing so.
  • Leanness. This goes back in part to the process stuff. It is a matter of keeping the process simple
  • Flexibility, team trust and commitment. We fall off the wagon sometimes. We need to be able to call eachother out on things so we can keep improving. We need to trust that we will stay committed and on track when thing fail – which they will.
  • Persistence. Stay the course when people push back on you. Managent, peers, and others.
  • Trust, with customers and sponsors.
  • Top cover, someone over the top that can allow you to be persistent and help cover challenges from further above.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: