IT chief warns about dangers of state’s archaic technology

New Jersey’s technology infrastructure is in desperate need of repairs, with critical state agencies that oversee unemployment insurance and motor vehicle services relying on archaic systems that are at risk of failing at any point and sending the state into disarray for weeks, according to a new report released by the state.

“All these legacy systems are supporting critical state functions, and the failure of any one of them would have immediate detrimental impacts on resident-facing services. Worse, the ability to quickly recover from a failure is in doubt as the aging infrastructure is more difficult to support and institutional expertise on the legacy systems is dwindling,” reads the Oct. 1 report.

Written by Chief Technology Officer Christopher Rein, the 14-page report criticizes previous governors for their lack of attention to infrastructure needs. It also stresses the priorities Gov. Phil Murphy and future state officials should make to drag the state into the 21st Century.

Among the top concerns is the state’s massive data center in West Trenton. Core IT and computing services are stored there and it hasn’t been substantially updated since it was constructed.

If the power and cooling systems there fail, “residents and state operations would be impacted with timelines to recover likely stretching into weeks,” the report says.

Rein’s underlying message: New Jersey’s legacy systems need a major revamp to bring them up to present-day standards.

Since Murphy took office in 2018, the state’s Office of Information Technology and other agencies have worked to modernize systems, including by improving cybersecurity and adopting a cloud computing system, according to Murphy spokesman Michael Zhadanovsky.

“Governor Murphy took office after decades of neglect of our IT systems infrastructure, and his administration is committed to improving and modernizing the state’s technology infrastructure for the benefit of New Jersey residents,” Zhadanovsky said.

The COVID-19 pandemic laid bare the crumbing nature of New Jersey’s technology infrastructure, beginning with the Labor Department’s inability to process a torrent of unemployment claims when businesses were forced to shutter in March 2020.

At one point, Murphy pleaded during a press briefing for coders expert in COBOL, a programming language created in the 1950s that hasn’t been taught in most universities since 2000.

And when Motor Vehicle Commission centers closed across the state, the agency’s website was not equipped to handle services typically handled in person, leaving people to camp out on long lines snaking around the offices once they re-opened. The MVC issues have been a key gripe among Murphy’s GOP critics, like state Sen. Anthony Bucco (R-Morris).

“We’ve heard these calls for a long time that the state’s IT system needs to be upgraded. And now, we witnessed what happens when you ignore these problems,” Bucco said in an interview. “You can’t kick the can down the road anymore, because it’s only going to get more expensive.”

Running on technology that no longer exists

The report delves into the outdated computer systems that need to be modernized, and notes some funding for upgrades could come from the $6 billion in American Rescue Plan money the state is still sitting on, known as the Coronavirus State Fiscal Recovery Fund (SFRF).

A full modernization of the Department of Labor’s unemployment and temporary disability systems would take two to four years to upgrade, the report states, with the first year of funding already allocated in the 2022 state budget. However, a major hurdle that Labor Commissioner Robert Asaro-Angelo has previously discussed is changes at the federal level that may require additional service upgrades.

Some of the MVC’s core database and applications were upgraded over the last five years, but there are still “substantial components” relying on old programming language and legacy databases, the report says, adding “a barrier to meeting the new and evolving needs to serve” New Jerseyans.

The Department of Treasury’s three highest risk and critical systems — budgeting, tax administration, and accounting and finance systems — are all slated to see some upgrades this year. Those are potentially eligible for funding through SFRF funds.

The Department of Corrections’ main computers are relying on an unsupported system, meaning the vendor who provided it is no longer in business. That system is also used by the Parole Board and Juvenile Justice Commission.

Other aging systems, including the state payroll system, which functions on the oldest single system, are relatively stable.

Bucco said the $14 million set aside to modernize the unemployment system isn’t enough to fix it, and doesn’t address issues with MVC or DOC. He plans to introduce a bill dedicating $200 million in federal stimulus funds to upgrade the state’s IT systems.

“Now that we have resources available, we need to use them to make these corrections. This is a real issue and a real problem, and the CTO has rightfully pointed out we need to be aware of a crash that could take weeks to recover from,” he said. “So much has to be done right now, and you’re just taking a chance that nothing major occurs that will shut everything down.”

Some of those modernizations are underway, according to the report. Phase One of a multi-year upgrade to the data centers — which was years overdue, according to the report — will upgrade electrical feeds and battery backup systems. The five-year project is estimated to cost $64.3 million, and needs to be funded annually to mitigate the risks of an outage.

Other critical upgrades were undertaken prior to the pandemic. Up until three years ago, state agencies had no access to cloud computing, which offers stronger security and eases accessibility for business needs, the report says. Two cloud environments were mostly configured by March 2020, allowing OIT to quickly adjust to pandemic response.

The cloud provides computing power and storage to agencies in hours, rather than weeks or months.

“Never had the state encountered the demands and challenges presented by a global pandemic; but never has NJOIT been able to respond so quickly with essentially limitless, scalable infrastructure to meet these demands,” the report reads.

Other upgrades included the use of multi-factor authentication to protect data; a highly-rated platform to stop breaches, malware, and ransomware attacks; and a network resiliency project to strengthen the state’s secure network. Most of these are underway, with projects continuing for the next three years.

Staffing issues

As part of the report, more than 23 state agencies responded to a survey of their critical needs and greatest barriers to entry. Overwhelmingly, agencies said recruiting and retaining IT staff is a major issue.

IT staff can make a lot more money in the private sector compared to working for the state. The lack of an attractive salary, the agencies say, is one of the biggest hurdles in retaining staff. It also leads to the loss of staff with institutional knowledge, according to the report.

“Although this shortage of IT talents exists across both private and public sectors, the impact is more pronounced, and therefore the resultant risk is greater to public sector enterprises, such as the state’s executive branch agencies,” the report reads.