Container Technology Strategy Hints at Change Management Across Government

Container technology — developing and packaging software with all of its related components so that it has a standardized deployment across all systems and servers — is not a new concept in IT, but has recently gained broad traction across enterprises in both the public and private sector.

Consider Lego blocks: irrespective of what set they come from, one can use them in multiple configurations, quickly building or rebuilding new models. Containers work much in the same way, allowing agencies to develop services as blocks of a larger system, and migrate from test environments to a variety of systems seamlessly.

“That’s why I love containers,” said Michael Rivera, director of server support and executive sponsor of services enterprise container platform at IRS. “They’re the same every single time!”

At USCIS, container technology was “a big, big push” starting four to five years ago, said USCIS Cyber Division Branch Chief Adrian Monza at the IRS Container Technology Inter-Agency Forum hosted by ATARC last month.

Monza explained how container technology helped USCIS manage its immigration application case management system, Ellis.

“[We] had a large, monolithic Java application, and it was getting really, really hard for lots of teams to develop on that single application,” he said, adding that this is a common obstacle those in both the private and public sectors encounter. “They kept breaking each other’s builds, nothing worked, deployments were a mess, and they were horrible and miserable and awful.” The solution, he outlined, was to take that single application apart into microservices.

“As we’ve done that, the net effect was to enable individual development teams to be able to build and deploy code autonomously from each other, so that way we can get features quicker to our end user,” Monza said. “That really is the ultimate goal of all of this.”

Underscoring this effort was the goal that drove the shift to containers and microservices at USCIS, Monza said. He estimated that USCIS is currently running 550 services and over 1,000 containers in its nonproduction environment, and 270 services and just under 500 containers in its production environment.

As with any change in how IT operates with an agency, this shift could not have occurred without an accompanying culture shift.

Developing mature pipelines at USCIS involved “some trial and error,” Monza said, along with adopting industry best practices in the space.

“When we started,” he said, “our pipelines were something we had developed fairly manually. Now, we’re even automating the deployment and operation of the pipeline itself — that’s coded as well.”

While reducing costs was certainly a factor in adopting container technology, Monza said, the main driver was reducing the time to market. Like many other enterprises, USCIS previously used a virtual machine platform to run software, but found that containers enable a much faster timeline from idea to production.

Although moving to container technology enables changes to software systems in a matter of hours, the overall shift required a new approach to development, orchestration software to manage the containers and microservices and a review of existing configuration and change management practices. Here, the lessons learned and best practices for container technology are applicable to change control and management across any sort of development or IT process.

“At [Federal Student Aid], we are in the process of maturing [those practices],” said Surendra Babu, systems security manager technical specialist of IAM and cloud migration at the Department of Education. “We are trying to make people comfortable with the baseline that we have established. That is the key. Establish a baseline, get everybody comfortable with the baseline, then your continuous improvements to that baseline [are] very easy to get across.”

Once the baseline shift to container technology was implemented, communicating smaller changes across multiple offices became much faster.

“All people are created equal, but not all changes are created equal,” Monza said. “There’s a big difference in types of changes, and your change control process absolutely needs to reflect that.”

He advised using a change control board to coordinate agency-wide or department-wide changes. The board has a large role in governance early on, but as development teams become more mature, it will step back.

“Figure out what it is that you want [your developers] to do,” Monza advised. “What are you trying to drive doing change control? Find a way to incentivize the development community to do it themselves … and that lets you focus your time, effort and energy on the things that matter. If you’re not going to sit down and have a discussion about a change, then why are you approving or reviewing it in the first place? Just get going.”