San Francisco Locked Out
Original post: 20 July 2008
Terry Childs, a network administrator for the City / County of San Francisco, was arrested last week on four counts of computer fraud. He presently sits in the San Francisco County jail on a $5 million bond. Childs apparently configured the City’s Cisco-based network so he along had the password(s) to control and manage that network. And – seven days after the arrest – the City’s Department of Telecommunications and Information Services is apparently still locked out of its network. The original report of this incident from the San Francisco Chronicle is here. Paul Venezia of Infoworld blogged “insider” information here which he obtained in an e-mail from a SF employee. Although this is an anonymous source, Venezia’s story certain rings true to me. (Note: Although I know Chris Vein, San Francisco CIO, and count him as a friend, I have not discussed this incident with him nor do I have any personal knowledge of the event).
This situation, on the face of it, is both outrageous and troubling. I won’t speculate about why it occurred in San Francisco, other than saying Venezia’s blog has the ring of truth. The larger question: has it happened elsewhere and could in happen again in another public agency or government? And what, if anything, can we do to prevent it?
Has it happened before?
Emphatically and undoubtedly the answer is “yes”. Can I cite a specific example? Not immediately, but there are many many networks – and too many of them are dependent on a single “guru” or talented individual. A couple of caveats are in order here: first, in San Francisco Childs only managed the data communications network of routers and switches – he did NOT have access to applications, databases, and servers. That’s why most City technology functions appear to be working fine. Second, most networks are owned by private companies and businesses. They are NOT in the public eye as the City of San Francisco or the City of Seattle. Security incidents in private networks or even smaller government networks will not be visible to the public or the press.
Could it happen again, elsewhere?
Again, undoubtedly it will. However, I think such an incident in a large network is quite unlikely. Such networks require a number of technical people to cooperatively manage. And the larger the network the more rigorous and formal change management processes are required. Indeed, according to Venezia’s blog, it was a requirement for documentation and change management which might have sent Childs over the edge.
Indicators
Several small points are buried in the news articles: first, Childs allegedly monitored management’s electronic mail. Most technical folks in most organizations have some ability to do this. But most public employees (in my experience) have much higher standards of integrity. And with the availability of e-mail encryption, good security monitoring tools, and teams of employees working together, such monitoring should be rare and declining.
Next, San Francisco recently hired a Chief Information Security Officer (CISO), who was actively investigating, monitoring and instituting stronger security policies. Again, this is another factor which probably led to Child’s discovery and arrest. In my personal experience, CISOs have rigorous integrity and concern for processes and policies which protect agency information from harm.
Finally, Childs appears to have a strong ownership of San Francisco’s fiber-wide-area-network, proud of its construction and reliable operation. These are noble attributes which I find in many public technology employees. He also apparently had a disdain for other administrators, staff and management in the department. This is, thankfully, a rare attribute in my experience.
How can we prevent future occurrences?
Some will suggest conducting “background checks” on employees. These are valuable. We’ve been doing them as a matter of course for five years at the City of Seattle’ DoIT. However background checks merely make sure we’re not hiring employees with a history of convictions for driving while intoxicated or a current set of 100 unpaid parking tickets. And they would not have prevented the Childs’ incident. More importantly, when hiring we need to look for employees who are personable and can work as members of a team. Smart employees can be trained for technical skills. In the distant past (1980s), technology employees were very proud of their programming (“networks”, “systems”, “code”), identified it, and defended it intensely (“there aren’t any bugs in that program – I created it and tested it – are you questioning my technical skills?”) Today we can’t afford that – we need employees who are proud of the technology they control, but who have a life and an identity outside of the work they do. Employees who build reliable systems, but realize that it is not the system which matters, but the fact that the 600,000 people of the City of Seattle are safer and happier because their government uses that technology to better serve them. And we also need to employ “best practices” in technology management, hire Chief Information Security Officers, and have employees and technically-astute management who are diligent with change management processes to keep our technology operating reliably.
A Personal Note
A couple of years ago we at the City of Seattle hired a new network administrator. His managers and I fired him after six weeks on the job. Indeed, we should have fired him after two weeks. He displayed a penchant for trying to hack into network switches rather than collaborate with others on the network team to manage them and administer them. The lessons: teamwork is the most valuable attribute in any public employee! You can train and educate folks to be technologists, administrators and managers. Training for teamwork is much harder – you need to look for it when hiring. Second: don’t hesitate to act on bad behavior. And for this, the management San Francisco’s Department of Telecommunications and Information Services should be commended, even if it was late.
the Chief Seattle Geek blog 
“Terry Childs, a network administrator for the City / County of San Francisco, was arrested last week on four counts of computer fraud.”
No, there is not a single charge of computer fraud against anyone in this case. He is charged with violations of California Penal Code subsection 502 “Unauthorized access to computers, computer systems and computer data”.
Specifically, one count of violation of 502(c)(5) and three counts of violation of 502(c)(6).
The DA’s complaint against Childs is available at the following URL:
Click to access tchilds_complaint.pdf
There is no person employed by the City and County of San Francisco who has the title “Chief Information Security Officer”.
“Interview for the Security Manager position starts Monday, February 25, 2008”
http://www.sfgov.org/site/coit_page.asp?id=79923
“Chairperson Robinson introduced DTIS’ new Security Manager, Jeana Pieralde.” — June 11, 2008
http://www.sfgov.org/site/coit_page.asp?id=84457&mode=text
“Chairperson Robinson announced that DTIS internally hired the new Security Manager, Jeana Pieralde.” — April 29, 2008
http://www.sfgov.org/site/dtis_page.asp?id=82542
“A couple of caveats are in order here: first, in San Francisco Childs only managed the data communications network of routers and switches – he did NOT have access to applications, databases, and servers. That’s why most City technology functions appear to be working fine.”
The City’s Cisco routers on the FiberWAN (which Terry Childs did have full control of) were also working fine throughout the entire spectacle. At no time did Terry Childs render any portion of the network non-functional even where he did not have mere access but complete and sole control and where he could have absolutely shut-down the network completely -but did not-.
“Again, undoubtedly it will. However, I think such an incident in a large network is quite unlikely. Such networks require a number of technical people to cooperatively manage”
Obviously not. The City and County of San Francisco had one, and only one, employee who met the minimum qualifications set by the City and was given responsibility to install, configure, maintain and secure the FiberWAN network. Terry Childs.
“Several small points are buried in the news articles: first, Childs allegedly monitored management’s electronic mail.”
True insofar as some reports have stated that he was accused of of doing so. However, he actually hasn’t even been accused of doing so nor is he charged with doing so.
From the Assistant District Attorney’s opposition to the motion to reduce bail (both paragraphs below are shown verbatim from the document at the following URL: http://weblog.infoworld.com/venezia/childs/tcoppositiontoreduce_bail.pdf ):
“According to the experts working on the network, the Defendant could have access to files and data of different departments. First, data travels on the network unencrypted and can be read and captured by anyone monitoring the network. The defendant could have captured and saved this information while he was monitoring the system.”
“The Defendant had programs on the network referred to as “sniffing programs” that were designed to identify certain types of data that was moving on the network. These programs could be directed to look for certain types of data on the network and downloaded them to his hard drives for later uses.”
Claims such as “could have access”, “could have captured”, “could be directed to look for” are not the same as actual charges (nor even allegations) from the District Attorney’s office.
“Next, San Francisco recently hired a Chief Information Security Officer (CISO), who was actively investigating, monitoring and instituting stronger security policies.”
Herein lies a bit of a quandary. Network Security Manager Jeana Pieralde was appointed to the newly created position just a few months ago. Her job is to create a security policy, submit it to the Committee on Information Technology and after it’s made official, to work with a group of security personnel from various departments to implement the policy. However, the policy isn’t actually official yet. Its still awaiting approval from COIT.
Thus with no policy to enforce or with which to determine compliance or lack thereof, it is a mystery what exactly Jeana Pieralde was doing performing an unannounced, after-hours “security audit” in a City office other than that in which she herself worked. It was during that secret “security audit” on the evening of Friday, June 20th, 2008, in which Jeana Pieralde took a hard drive from another City employee’s office and was photographed by Terry Childs as she did so.
The office from which Pieralde removed the hard drive belonged to DTIS Security Officer Nancy Hastings (who naturally was not present in the office because the “security audit” was being conducted after hours.)
Terry Childs had returned late to the offices (which do include his office and do not include Jeana Pieralde’s office) at about 5:15 P.M. to find Jeana Pieralde (who does not work in those offices) taking a hard drive from one of Terry’s co-workers offices. Terry photographed this act with the camera in his cellphone.
Jeana Pieralde then involved DTIS Deputy Director Rich Robinson. Rich called Terry and told him to stop taking pictures.
Three days later (Monday) both Rich Robinson and Jeana Pieralde filed complaints of threats with the San Francisco police department and Police Inspector James Ramsey was assigned to the case. No charges have ever been filed against Terry Childs for the alleged threats (which included the statement “I’m ready for you Rich. Or I can come up to your office.”)
However, Inspector James Ramsey did stick around and investigate Terry Childs, culminating in his arrest for violation of California’s “Unauthorized access to computers” law (Cal Penal Code subsection 502.)
Which presents another quandary because it appears that the only employee of the City who had clear authorization to administer the routers and set passwords was Terry Childs. Thus “unauthorized access” seems at the very least, a suspicious charge.
Further, even the draft of the City’s security policy clearly and repeatedly states that passwords are to be kept confidential, not divulged, not even to co-workers or one’s boss and that the only the head of the department (Chris Vein in DTIS’s case) as the “information owner” has the authority to determine who may have access to any password.
Neither Inspector Ramsey, nor the Assistant District Attorney Conrad B. del Rosario, Jr. have claimed that Chris Vein ever asked Terry Childs for a password or for any information at all.
It certainly will be interesting to see this case continue.
Please identify who you are. Herb Tong denies that it is he who posted these replies.
R. Shikman
Attorney for T. Childs
Maybe, everyone should step back and let the court decide this case. Terry Childs is getting due process. A judge found reasonable evidence of the intention of serious harm to the city of SF to hold Childs.
Arguing about what happened or what charges are filed is a legal issue. Laws require specific aspects to be proved by the prosecution for a conviction. No one can say at this point whether or not the prosecution can convict. OJ SImpson got off because a lawyer convinced some jurors of reasonable doubt. It was a travesty of justice, but we live in a democracy where most criminals are not convicted.
But the techies that argue over the law on this. It is clearly a case of harm. You can’t say that because you only “kidnapped” and didn’t harm someone or an entity, that it is not a crime. It clearly is. The potential of damages from someone pulling a stunt like can be monumental. If you hijack an airplane, and it lands without anyone getting hurt, you are still guilty. You can make this analogy to this case a hundred ways, and it always comes down to Childs was wrong. I believe he is clearly deserving of a conviction, but I can accept whatever the court decides. I am not on a jury listening to the case. It has to be left to the judge.
Childs has all sorts of extenuating circumstances that made what he did awful; packet sniffers, passwords. The fact that he is only being pursued on a few of the charges is a legal limitation.
And to the earlier comments, of course cities need better peer review and management. But if you have ever seen the limits of resources in government IT, you can see where these things are so easily abused by rogues who think they own something or are doing a government entity a favor by hacking them or locking them out.
The bottom line for me is these threads keep trying to build sympathy for Childs as some sort of innocent martyr. He is not. Maybe a conviction and commuting to time served is a fair outcome, but really he owes the city a great deal in damages. Not for his incarceration, but in the fact that by manipulating the systems, they and almost anything that touched them, including database or other servers would need to be completely rebuilt to insure backdoors were removed. Take his pay, until it is payed off. If he showed true remorse and willingness to cooperate in the beginning, would he have ever gotten into this legal corner? No.