Los Angeles IT secures the vote with open source and the cloud

Vital for democracy, voting system integrity has come under increasing scrutiny of late, placing pressure on government IT leaders to ensure election systems are robust and uncompromised.

Aman Bhullar, CIO of Los Angeles County Registrar-Recorder/County Clerk, has heeded the call, having led a widespread overhaul of antiquated voting infrastructure just in time for the contentious 2020 presidential election — a transformation rich in open source software with a mission to help ensure other counties can benefit from his team’s work.

Partnering closely with AT&T Cybersecurity, Bhullar and his IT team deployed a mix of advanced security, open-source software, and the AWS cloud to build its venerated Voting Solutions for All People (VSAP) platform, a project that consists of more than two dozen individual systems.

“We started thinking about the election infrastructure about 10 years ago because we knew it was aging and with our county’s scale and complexities, we really could not find anything out of the box or any election technology vendor that could meet our needs,” says Bhullar. “That’s when we started with human-centered design.”

With more than 6 million citizens, Los Angeles county is among the largest in the US, and Bhullar has served as county clerk CIO since 2016. He and his IT team of about 250 IT pros worked with AT&T Cybersecurity to secure the new systems and with contract manufacturer Smartmatic to build the voting machines. The team also developed myriad open-source applications running on CentOS to enable the county to share VSAP’s highly secure, top-notch voting infrastructure with other jurisdictions.

“The core principles of VSAP were that even after we go live, we wanted to make our code base open source because we want to help the other jurisdictions so that they can benefit from the investment that we made,” says Bhullar, whose team has developed an open-source plan and is working with the State of California to establish a framework for making its custom code open.

For that open, secure voting infrastructure, Bhullar and his team have earned many awards, including a 2023 CIO 100 Award for IT innovation and leadership.

Securing the vote

No doubt, security is the major pillar of VSAP.

But aside from that crown jewel, Bhullar’s team developed a stack of open-source software for voting services and deployed the AWS cloud for constituent-facing services and voting information. While still working with the State of California to make its custom code public, roughly 70% of the multifaceted election stack is open source, with the remaining 30% proprietary.

AWS services have become a staple for LA County, especially throughout the pandemic. Several of the county’s 40 government agencies used Amazon’s AppStream 2.0, for instance, to facilitate remote work during the pandemic, including the LA County Department of Public Services and the Department of Children and Family Services. And, in advance of the 2020 Presidential Election, Bhullar and his team turned to Amazon to migrate existing .NET citizen-facing websites to AWS to provide voter outreach and constituency services as part of VSAP.

This move to the cloud made the county’s election infrastructure more scalable, secure, and available with access to all the information citizens need to vote, such as the location of polling stations, the ability to view results, and links for disabled citizens to request assistance such as mobile voting vans, Bhullar says.

In the end, voting remains a manual process with physical ballots that are scanned, stored, and closely guarded in traditional boxes, Bhullar emphasizes. “The actual voting experience is not cloud-based. It is not connected to the outside world,” he says. “It is in isolated networks and the devices are not connected to any external networking during the voting period.”

The county’s cybersecurity program — which guards against misinformation and disinformation — remains top of mind in all elections. The first component, E-CSOC, was built by the Los Angeles Registrar/Recorder and County Clerk security team in conjunction with AT&T Cybersecurity. It provides early detection of threats that may jeopardize elections and enables the combined security team to monitor the strength of its security controls.

“During major elections, the E-CSOC operates 24/7 for the duration of the 11-day voting cycle with onsite and remote support to monitor for any potential security threats or indicators of compromise,” according to documents supplied to CIO.com by LA County. “The E-CSOC collects security data from multiple on-premises and cloud environments and correlates events across all assets, presenting correlated alarms in a single pane of glass.”

The cybersecurity program also includes identity and access management; third-party penetration testing; governance, risk, and compliance assessments; and endpoint monitoring.

To accomplish the highest security level, VSAP leverages data from state, federal, and international security agencies, including EI-ISAC, FBI, and CISA as well as the Open Threat Exchange (OTX), known as the largest open threat intelligence community on the globe, the county reports.

Bhullar also notes that the LA County cybersecurity unit operates a social media monitoring platform in league with the federal government and has enlisted some cooperation by large social media companies to prevent the impersonation of office bearers.

One analyst noted that while there has been no widespread election fraud in the US, it is important for all election officials to safeguard the voting infrastructure.

“The LA County initiative sounds like a great modernization of IT infrastructure for the current realities of cyber risks for state and local governments across all functions, but there is no research to indicate that election areas such as voter registration or election night reporting or voting machines have been compromised or more targeted than other systems,” says Ruthbea Yesner, vice president of government insights, education, smart cities, and communities for IDC.

“That said, national foreign actors do target state and local government systems, which do need to continually improve cybersecurity and resiliency to ensure public trust, protect the public, and combat misinformation campaigns,” the IDC analyst told CIO.com.

Bolstering network operations to facilitate the vote

VSAP’s second pillar — E-NOC, the county’s network operations center — enables the IT team to quickly deploy and support network connectivity to more than 1,000 voting locations across a 4,000-mile-wide area. It also offers election operations management and Vote Center connectivity, as well as support for technician services, according to documents from the county, which views E-NOC as a cornerstone of the value of VSAP.

“In a very short period of time, I have 1,000 pop-up networks throughout the county that need to be secured and redundancy built into the network connectivity,” Bhullar says. “Imagine in a building there is an isolated secured network. Scanning is part of the tabulation process in an isolated or air-gapped environment. It never leaves the environment.”

LA County works with all telecommunications carriers and satellite companies to ensure there are no lapses during the election period, Bhullar adds.

“We do penetration testing before we even institute the pop-up networks at these 1,000 locations. We assess the network to download/upload speed, and we also have redundancy built in so our routers support up to four carriers when it comes to cell phone connectivity. If TNT is giving me a good connection and it drops, I can quickly switch to T Mobile, for example,” he says.

But the beauty of LA County’s election infrastructure, Bhullar says, is the homegrown nature of the voting machines and the voting applications. The ballot marking devices developed by LA County, for example, record the voter’s input and converts it into a “human readable ballot,” the CIO says, noting ballots are retained in an Apache Cassandra database management system for a legally defined period.

Bhullar’s team developed several open-source applications, including a vote tabulation system that counts and scans mail-in ballots and in-person votes coming from voting centers. The system, based on CentOS and Cassandra, is not connected to external networks but has access to the voter registration database. Once regulatory go-ahead from the State of California is in place, Bhullar’s team plans to make this and other custom developed open-source code available to the public.

The IT team also developed an interactive sample ballot application to educate constituents about the ballot and provide clear explanations of the legislative proposals confronting voters.

Finally, the IT Enterprise Command Center, of which E-CSOC is a key part, offers a call center with responsibility for minor and major incidents. For example, in the September 2021 California gubernatorial recall election, ITECC was able to proactively address more than 800 health check tickets and resolved all 88 major incidents reported to the system, Bhullar’s group reports.

Ensuring future voting integrity

Going forward, Bhullar intends to keep evolving VSAP to make voting easier and more secure. For example, the CIO is currently in the process of moving from CentOS to Red Hat Linux to gain access to Red Hat’s deep open source portfolio, including multicloud support.

AI will also be a factor in future elections as it has been in recent elections, says Bhullar, noting that the county inherited AI tools following an undisclosed vendor acquisition and used them to gain insights on voting patterns.

“There is a component of artificial intelligence here as well so that we can have machines take a look at the patterns that are coming in,” Bhullar says. “In a large election, there are about 600 million events that are happening. Obviously, it is not feasible for humans to look at this. I would need an army of people to look at all these events.”

The CIO is concerned about the possibility of bad actors using generative AI to spread disinformation about candidates and policies but is equally confident in the security team assembled to fight off these attacks emanating from social media outlets and even the possibility of an inside attack.

“I think we can deal with the traditional threats or the threat vectors, such as state actors from outside the country attacking us,” he says. “The good thing is our voting systems are owned and controlled by us and they are not connected to the outside world. So, I think there’s an inherent security and confidence there.

“I think we’ve got it covered,” the CIO says.

[Editor‘s note: Amendments were made to this article after publication to clarify that the Los Angeles County Registrar-Recorder/County Clerk is working with the State of California to establish a regulatory framework for open-sourcing its custom code, which is not yet available to the public.]