Closed Loop Verification: A Must-have for Network Automation
How could closed loop verification benefit your network automation journey?
Closed loop verification may be the answer to the threats and vulnerabilities that plague network automation. Fabrizio Maccioni, director of technical marketing at Forward Networks, takes a closer look at how your network automation journey risks could be tackled better with closed loop verification.
Not too long ago, IT professionals physically connected to network devices for upgrades and changes. Imagine trying to physically go to every device on your network, plug in and manually push an update. It’s obviously an impractical and unaffordable proposition that would require a veritable army of IT professionals and weeks, especially for organizations with data centers spread around the globe. Undoubtedly devices would be missed, and other mistakes would be made, increasing the risk of outages or vulnerabilities.
The Risks of Network Automation
It’s clear that network automation is no longer a “nice to have” or “cutting edge” technology for global enterprises, it’s simply a requirement for managing and maintaining today’s networks. Automation enables IT pros to push updates to the entire organization with one click. While this is a compelling time saver, it can also introduce significant risk to the network by pushing non-compliant changes. Automation must be implemented responsibly.
The benefits of automation outweigh the risks by far. Automation is part of a toolset that helps NetOps teams keep the network up to date and deploy new services faster and more reliably. However, it must be implemented responsibly in a way that minimizes risk to the organization, mitigates the propagation of human error, and protects the behavior and security posture of the network.
Savvy network engineers are beginning to employ closed-loop verification for network automation. Using this methodology provides a safety buffer by comparing the network state before changes are pushed with the expected state. Powered by digital twin technology, closed-loop automation is the insurance policy administrators need before (and after) pushing automated changes live. Imagine being able to push the button without anxiety.
How Closed Loop Verification De-risks Network Automation
There are several tools on the market that help administrators build automation frameworks. Typically, this involves:
1. Creating the service request – what is required
- Developing the service definition recipe – creating the code to solve the problem
- Use of an orchestration tool to push the configuration – the change goes live
There’s an important element missing from this workflow; assuring the changes will behave as intended. Network automation empowers engineers to write changes once and push them out to the entire network with the click of a mouse. But what if the code has an error in it?
Until today we’ve had to depend on laborious and error-prone human code reviews to ensure changes are error-free and policy compliant. Have you ever read a published book with one or more typos? Most books have typos even after an extensive editing and proofreading process. The same is true for code, except when we encounter a typo in a book, we smile and read on. A very simple coding error can take down the network. Human review does not provide the level of assurance that mission-critical networks need. Network automation needs network verification to be successful and safe.
Closed loop automation means verifying the network state before and after changes are pushed to make sure no side effects are introduced by the changes and rolling back to a previously known working state in case of any issue.
See More: Intent-Based Networking: What Is It and Will It Usher an Era of Agile Networks?
How It Works
Technology is available to make it easy for network operators and engineers to automate their networks. In fact, today, NetOps and engineers can drag and drop apps, such as software upgrades, network device and firewall configuration changes, device onboarding, and more, into the network to build in automation quickly. Pre-built automation tools eliminate manual tasks, increase the velocity of changes, and mitigate network downtime due to changes.
Once the desired automated tools are in place, the NetOps team sets the appropriate permissions for each automated step and then exposes permissions to those in the organization who have access to run the automated steps throughout the network. With closed-loop verification, while each change is deployed, the network constantly verifies accuracy and monitors for compliance to ensure it is behaving as desired. By merging network automation with network verification, this closed-loop process allows NetOps teams to deploy changes at scale and avoid unpredictable side effects.
Using closed-loop automation, administrators can verify routing, test the efficacy of new security rules, verify new service connectivity, and check for side effects. In short, the process enables administrators to collect input, check for efficacy, test connectivity and security, deploy the change, and verify it’s behaving as desired.
This is a sample workflow to fully automate a new service deployment:
Input: Clone the repository that stores the automation playbooks and the automation workflow description file (e.g., Jenkinsfile if using Jenkins as the automation orchestrator). User input will be requested (in this case, the operation engineer), who’s checking for IP addresses, user location, port, and the name describing the new service. This is the last time the workflow requires manual input. The system will alert users of the proposed updates via messaging tools (e.g., Slack, MS Teams, WebEx) and update the service ticket (e.g., ServiceNow, Jira).
Check: The check phase is essentially a path analysis to verify routing is as planned and conducting a hop-by-hop analysis to verify if one or multiple firewalls are blocking the path. The path analysis is saved to verify the service is working as expected every time a new network collection will occur. If no firewalls block connectivity, there is no need to deploy any change, and the automation is closed successfully. If there are firewalls blocking the path, we move to the test phase.
Test: Ideally, administrators can use a network digital twin to test the firewall rules in a sandbox to simulate a full production environment using a prediction feature. Alternatively, administrators test the rules in a pre-production environment that represents only a subset of the network. The test is intended to ensure connectivity and identify any unintended side effects. If the check fails, the workflow is reverted to engineering for repairs. If it passes, we move to deployment.
Deploy: Using the automation tool of choice, the update is pushed live throughout the network.
Verify: Collect configuration and state data on the devices updated and verify that they are behaving as intended. If there is an issue, the configurations can be rolled back using the configurations collected before the change was pushed live. If the change is successful, users will be notified.
The Benefits
Employing closed-loop verification to automate complex, business-critical changes within on-premises, hybrid, and multi-cloud networks assures network operators that any changes made in the network have the intended effect and that no unexpected behaviors are introduced when automating network changes.
This is critically important to maintaining a strong security posture and compliance. The consequences could be devastating if a change exposes any part of the network. Closed-loop verification with automation makes complex networks more agile, predictable, and secure.
Do you think closed loop verification could reduce the risks often faced in network automation? Share your thoughts on Facebook, Twitter, and LinkedIn. We’d love to hear from you!
Image Source: Shutterstock
MORE ON NETWORK AUTOMATION:
