- On Thursday, several customers of the IT automation startup Chef faced outages after a former employee removed code from his account that Chef's software had been relying on.
- Seth Vargo, the former employee, decided to do this after finding out that his code was being used by Immigration and Customs Enforcement, which Chef has a $95,500 contract with.
- Chef CEO Barry Crist wrote an email to the company saying that he and the leadership team made "a principled decision" to work with government institutions, regardless of whether they personally support their policies or not.
- Chef's teams have since restored service to its customers.
- Visit Business Insider's homepage for more stories.
When open source developer Seth Vargo found out that his software was being used by the US Immigration and Customs Enforcement, he decided to make his disapproval known.
Vargo previously worked at Chef, an automation software startup, and he had been hosting Ruby Gems — an open source packages of code and documentation — on his personal account. Chef's software had been relying on Vargo's Ruby Gems for its software to run.
Vargo, who left Chef in 2014, recently learned that Chef had a contract with ICE through Twitter, and then he verified it on USAspending.gov, as well as with current and former Chef employees.
"I discovered that my code was being included in a distribution used by ICE, which I personally believe to be evil," Vargo told Business Insider.
So on Thursday morning, Vargo removed several Ruby Gems from his own account.
That resulted in an outage for some of Chef's customers because Chef's software depended on these gems being available. Chef had an all-hands meeting on Thursday afternoon, and the team worked to restore services for its customers.
Now, Chef CTO Corey Scobie says, the Ruby Gems have been restored to their original state, and as a company, Chef is focusing on its customers' health and success.
The tech industry's ties to ICE has already caused controversy and turmoil for companies like Palantir, Microsoft, Amazon, and Salesforce, as company employees and outsiders have protested and called for contracts to be scrapped.
But the incident with Chef reveals a new, perhaps more intractable risk for tech companies whose products often rely on building blocks made of open source software that's outside their control. As Chef discovered, a company's employees are not the only group with the power to react to controversial business practices — and even a small change to an open source component can create ripples that cause chaos for the business.
"Yesterday we were faced with a significant and serious customer event that was the result of actions that were taken outside of our immediate span of control," Scobie told Business Insider. "We weren't consulted before those actions were taken. Our employees rallied around putting our customers first and making sure we repair the damage that was done to the employee ecosystem."
'This decision is not about contract value'
Chef CEO Barry Crist wrote in a blog post and email to the company that Chef started working with ICE during the previous administration "to modernize their IT practices." Regardless of whether they personally agree with their policies, Crist says he made a "principled decision" with the support of the Chef executive team to work with government institutions.
"I want to be clear that this decision is not about contract value — it is about maintaining a consistent and fair business approach in these volatile times," Crist wrote. "I do not believe that it is appropriate, practical, or within our mission to examine specific government projects with the purpose of selecting which U.S. agencies we should or should not do business. My goal is to continue growing Chef as a company that transcends numerous U.S. presidential administrations."
Crist also wrote that he disagrees with such policies, saying, "I also find policies such as separating families and detaining children wrong and contrary to the best interests of our country." Scobie echoed these sentiments.
"I think generally speaking, on an emotional level, a lot of people, Barry and myself included, are quite distraught with what's happening with ICE in particular with the actions of the current administration in our government," Scobie said.
Chef had been working with ICE since 2015. Scobie says that originally, the Department of Homeland Security was using Chef's open source software before becoming a contracted customer.
"For context, we began working with DHS and ICE during the Obama administration to modernize their IT and to the best of our knowledge, no Chef software is being used in systems that further the separation of parents and children at the US border," Scobie said.
According to USAspending.gov, ICE awarded a $95,500 contract to C&C International Computers & Consultants, which uses Chef software licenses. This year-long contract began on Aug. 29. Chef had previously had another contract with ICE that started in 2015.
Scobie also says that in the past, the company had several discussions on its position around ICE's contract, and Chef decided to accept this contract. He says employees have had mixed opinions.
"There's lots of opinions about both the correctness and the business relationship and also the ethics and the moral angles to it," Scobie said. "One of the things we value at Chef is to have open, anonymous dialogue, and that's certainly something we attempted to have on this particular topic."
'Software used for evil'
Before yanking the Ruby Gems, Vargo said that initially, he explored options for changing Chef's software license. However, open source licenses allow people to use the software however they want, and groups cannot be prohibited from using it.
Vargo says that since he removed the Ruby Gems, Chef had not contacted him.
"I don't feel comfortable having my name and software used for evil," Vargo said. "...I think the community's response echoes louder than any words I could provide."
As for Chef, Scobie says the team is evaluating vulnerabilities and the code it depends on to make sure something like this doesn't happen again.
"Our goal will be to reduce that footprint and close that gap as much as humanly possible," Scobie said. "That's our goal to provide a stable computing environment..."Ultimately at the end of the day, we're depending on many many downstream things that are out of our control and could have vulnerabilities. That's the nature of open source. It's a web of dependencies."
Although working with ICE has been controversial in the tech industry, Scobie says he's not worried about it impacting Chef's business.
"I think we have seen no evidence at this point to suggest that there's a business risk at play here," Scobie said. "Our customers are extremely happy with our response to the outage that was caused yesterday and how we've been dealing with it in the past 24 hours."
Do you work at Chef? Got a tip? Contact this reporter via email at rmchan@businessinsider.com, Telegram at @rosaliechan, or Twitter DM at @rosaliechan17. (PR pitches by email only, please.) Other types of secure messaging available upon request. You can also contact Business Insider securely via SecureDrop.
Join the conversation about this story »
NOW WATCH: Octopuses are officially the weirdest animals on Earth