EPA Committed to Finding a Workaround for Every IPv6 Challenge [Archived]

By Jennifer Bly

EPA IPv6 Case Study

The Environmental Protection Agency (EPA) has made great progress toward the Office of Management and Budget (OMB) IPv6 goals for federal agencies. Brian Epley, the Office Director for the Office of Information Technology Operations, and his team at the EPA walked me through what it took to make IPv6 deployment a reality.

Key requirements for IPv6 deployment

First things first, Brian told me, “It’s important to understand the scope of our IPv6 effort. It was a multi-year activity, and I’m happy to report that EPA is going to claim a win.”

EPA is comprised of 10 regions and 14 program offices which are interconnected via a high-performance telecommunications network.  Employees utilize and depend on roughly 55 public websites, 1,900 servers, and 21,000 desktops. Because IT is largely decentralized across the agency, IPv6 deployment required senior leadership prioritization, a documented deployment approach, appointment of an IPv6 champion, and active participation from stakeholders in all regions and program offices.  These elements were essential for obtaining buy-in, minimizing re-work, ensuring no impact to daily operations, and maintaining the agency’s IT security posture.

Senior leadership prioritization

“Having a CIO make IPv6 adoption a priority across the agency allowed us to be successful,” said Brian. Strong support from the office of the CIO forced stakeholders to come to the table, to get new equipment installed and servers configured. Management’s keen interest both at a granular level and holistically was imperative for achieving a high level of interest and prioritization across the agency.

Let’s talk budget

Brian and his team recognized they would need to do a realignment related to ongoing initiatives within the existing budget. Despite OMB mandates and full leadership buy in, there was no new money earmarked just for an IPv6 initiative. Furthermore, in achieving the OMB requirements, they found themselves with net new initiatives, programmatic or organizationally driven, that require an additional level of effort and scrutiny. Looking at their as-is state, they made modifications where appropriate to push those things forward in more rapid fashion. For example, when the OMB mandate came out, it aligned with a preplanned telecom equipment hardware refresh at the WAN side for which EPA has a managed service. Due to this fortunate timing, they were able to include IPv6 requirements into that contract without waiting for new funds.

Taking a waterfall approach

EPA took a traditional waterfall approach to accomplish the many tasks required to deploy IPv6 within the agency. They scoped everything out so they could minimize rework and resource expenditure. Brian emphasized, “The planning piece gave us the metrics to show that we were purposefully and deliberately moving forward with the implementation.” A high-level outline for the EPA’s approach included:

1. Plan, implement, and validate core infrastructure for IPv6 readiness.

   a. EPA partnered with Department of Labor (DOL) on their IPv6 addressing plan and used it as a template to customize EPA’s approach.  This information-sharing helped them start further along than if they had to do it alone. EPA identified core infrastructure that was not IPv6 ready (EPA-managed and MTIPS-managed) and pursued getting the core infrastructure ready for IPv6.  Their final step was to validate the core IPv6 readiness.

2. Inventory, fix or replace, and test endpoints IPv6 readiness.

   a. EPA assessed what in-scope IT assets would require reconfiguration, updating, and/or replacing.  A representative sample of assets were used to test and validate IPv6 functionality before deploying agency-wide.

3. Verify monitoring and compliance tools functionality worked for IPv6.

   a. EPA inventoried all applicable IT tools used to manage and support monitoring functions and verified IPv4 and IPv6 functionality were on par.

4. Communicate findings, provide training, and report progress.

   a. EPA held weekly meetings with all stakeholders to report on progress, share lessons learned, and to identify training needs.  They prioritized information-sharing by creating a repository for solutions and suggestions.  They also provided periodic progress reports to agency senior leadership as well as OMB.

Technically speaking, the approach EPA took from start to finish was to make the core IPv6-ready and then work back to the end devices. This way, they were able to see the progression and make sure one part was operating appropriately before moving onto the next part. Brian and his team went in phases and used representative samples. They identified subsets that represented the use cases they thought they were going to have. Then they had technically proficient individuals apply IPv6, test out the functionality, and identify any issues. This process allowed them to deploy IPv6 in phases so most of their user base didn’t even notice a change.

Graphic provided by EPA

Challenges and Workarounds

“Whenever we ran into a technology issue that didn’t give the same level of support in IPv6 as IPv4, instead of taking it as a challenge we couldn’t overcome, we found a workaround for it—from the desktop up to the network,” Brian explained. Here are few challenges EPA faced and the workarounds they devised:

Challenge: EPA’s MTIPS provider lacked the ability to monitor and alert on IPv6 protocol failures.  When they first got IPv6 up and running on the network, there weren’t any tools available to identify when one of the two protocols were down. Either the circuit was up or the circuit was down. This was a big problem for EPA because when EPA had a high percentage of its clients operating on IPv6, the network operations team had to depend on user complaints to respond to critical network errors.

• Workaround: They had to work with their provider to come up with a nonstandard method for monitoring the health of the IPv6 protocol to make IPv6 errors more visible to improve the user’s network experience with IPv6.

Challenge: EPA required a refresh to its network switching hardware to IPv6 compliant devices. The timeline for procurement and deployment of new switching hardware forced the EPA to function with some network switches that were not fully IPv6 capable.

• Workaround: They suppressed DHCPv6 advertisements to networks that were forced to function on old hardware so the IPv6 deployment could continue while awaiting deployment of the new IPv6 capable switches.

Challenge: In general, EPA was faced with vendor products that did not have full IPv6 parity with IPv4.

• Workaround: In each situation, they worked with the product vendor for a patch or solution. For example, they worked with the agency’s cloud email service provider extensively to get IPv6 up and running, from doing packet captures to asking them to turn off their IPv6 DNS in order to access email during an IPv6 protocol outage.  “We continually had to push vendors to provide additional IPv6 support,” Brian said. “We had to convince them this was not only beneficial to us, but made a good business case for them. They could utilize our agency’s IPv6 experience to provide a better IPv6 service to other customers.”

Brian mentioned it is important to note that not every IPv6 problem that EPA encountered was an EPA problem.  Many times, they had to partner with technology vendors, external organizations and/or other federal agencies to resolve IPv6 issues.  This was necessary to ensure EPA users relying on that organization’s services were not inclined to disable IPv6 and stymie IPv6 deployment. “The general perception is that IPv6 is a nicety, not a necessity,” said Brian. “In many cases, if there is a problem, the preliminary response may be to turn off IPv6 to resume normal operations under IPv4.  It will take patience and persistence to get IPv6 accepted and adopted by the support community given the pressures of maintaining a 24 x 7 operational network.”

OMB IPv6 Goals Status

For EPA, IPv6 deployment was a multi-year effort that easily took 3-5 years if you include preparatory work required to meet OMB mandates.  The dynamic nature of IT coupled with on-going deployments of multiple initiatives caused fluctuations in EPA’s OMB IPv6 implementation progress metrics.  However, they have made significant progress on each of the three OMB IPv6 goals listed below:

FY2012 Goal: Natively implement IPv6 on public services (websites, DNS, mail, etc).

Intent: The general public is able to access US Government (USG) citizen services regardless of whether they are on IPv4 or IPv6.

• EPA was once 100% compliant with the 2012 mandate.  However, web failures for cloud-hosted domains occurred because cloud service providers were not yet fully supporting IPv6.

FY2014 Goal: Implement IPv6 on all internal Internet capable systems so that they can natively communicate via IPv6.

Intent: Ensure that the USG is able to do business with the general public regardless of whether they are on IPv4 or IPv6.

• EPA manages 21,173 desktops of which 21,115 of them are dual-stacked to support IPv6, putting EPA at 99.7% in compliance with the 2014 mandate. Even above and beyond the mandate, EPA implemented IPv6 on 70% of its servers, many of which were not required to have IPv6 on them for the FY2014 mandate.

Ensure IPv6 capabilities in IT acquisitions: Procurement.

Intent:  Leverage technology refreshes to secure IPv6-capable IT infrastructure.

• EPA scores itself at 92% compliance when it comes to IPv6 IT acquisition readiness. Thus far they have identified points of contact, issued agency IPv6 compliance memos, developed contract clause specification language, determined which forms, tools, and processes require IPv6 compliance, and implemented a procedure that would require a waiver from the CIO should an acquisition not meet IPv6 specifications. Now they are internally reviewing remaining documentation for baseline IPv6 requirement and anticipate full compliance before the end of the 2018 fiscal year.

Life-saving benefits of IPv6

What motivated EPA to adopt IPv6 was realizing not only the necessity for the mandate, but also the real-world application and benefit of IPv6. “We had onsite coordinators and emergency responders that could not deliver the mission without the effective application and use of IPv6. In my opinion, that’s what brought the passion of the team and success of this effort to the forefront,” said Brian.

EPA first responders encountered a problem where they needed to access EPA GIS data but their mobile devices were on networks that only supported IPv6.  Since EPA’s network was fully IPv6 capable, the problem was quickly resolved by configuring EPA’s local proxy device to listen on the IPv6 stack. Without IPv6, the outcome could have been dire. Take for example, the recent wildfires in California or hurricane season last year. Brian said, “We were able to accelerate and increase the number of human resources that were responding to the emergency and deploy mobile devices and other GIS devices to allow them to do their jobs.”

Advice for other federal agencies

Considering the success EPA has had with IPv6, I asked Brian and his team what advice they would give to other federal agencies that are in the process of deploying IPv6. They had several great recommendations including:

• Spend the time to think through and document an IP address schema that minimizes re-work.

• Develop an implementation plan that can assist with correctly sequencing activities and progress reporting. Also allow room for refinements.

• Get senior management (CIO) buy-in and support, because it is critical for success.

• Keep stakeholders engaged with regular progress reporting to management.

• Update key documents and procedures to ensure IPv6 becomes part of normal IT operations and maintenance activities.