DevOps: Plenty of Devs, Not Enough Ops

In spite of all the high-profile breaches that seem to sweep the headlines with greater frequency, companies slowly but surely have been getting a handle on internal security practices. At this point, it’s hard to imagine any employee, in or out of the tech sector, who hasn’t been run through antiphishing training.

However, security is only as strong as its weakest link, noted David Bryan, a penetration tester and senior managing consultant at IBM X-Force Red. The link that still needs reinforcing is also the one that — for a company marketing software products — is the most fundamental: developers.

In his presentation at the third iteration of the CypherCon hacker conference held last month in Milwaukee, Bryan described an anonymized engagement in which he probed the network of a development team responsible for 1.2 million user accounts. His purpose was to demonstrate that it is precisely the singular emphasis on developers speeding their code through production deadlines that leads to glaring security oversights.

“They have a deadline that they have to meet. The deadline doesn’t necessarily have to include security,” he said, but “it definitely includes functionality, and a deadline can mean the difference between actually taking a vacation and not.”

The deficit of security in development practices is due to more than just tight deadlines, though. Many developers can’t put security into practice because they never learned it in theory. There is such a dizzying array of concepts, languages, and tools for developers to get the hang of that often security and even basic networking concepts are crowded out of the curriculum in favor of more programming tradecraft.

“Even in these developer bootcamps, they’re just trying to get people up to speed on using the dev tools and not necessarily even talking about security,” Bryan said.

Hurtling Toward a Deadline

Programming has become such an indispensable tool that before educators have a chance to instill security consciousness in their trainees, they’re on to the next crop of students.

Referring to the infamous Steve Ballmer rant to which his talk’s title, “Developers. Developers, Developers,” cheekily nods, Bryan said, “We keep coming back to that. We need to get more people developing, which is good, but we forget about adding in security or adding in review of the environment, until a pentester comes along and says, ‘oh, hey, your machine is vulnerable, and it’s been vulnerable for X amount of months.'”

The final leg that props up this edifice is the prevalence of tools that — by their failure to require better security models — indulge the bad, if understandable, habits of twitchy developers hurtling toward a deadline without the background to know what, beyond functionality, they should be looking for in reviewing their work.

“Why are [DevOps tools developers] creating tools, like Jenkins or Marathon, that don’t require authentication? Just because it’s behind a firewall doesn’t mean that some attacker isn’t going to actually try and leverage it at some point,” Bryan pointed out.

In a way, this component is a natural outgrowth of the preceding one, in that developers of development tools on rigid timetables and lacking a sense for security will create tools that embody those traits, only to perpetuate the cycle when developers in the rest of the software world depend on them in their work.

A Little Goes a Long Way

So how does the industry treat these development ills? Like any malady, treatment starts with diagnosis.

“I would say it’s probably 50/50: I think there’s some onus on app-dev type tools to actually create logins, provide logins, things like that,” Bryan said, “but I think it’s also on the development team too, from the perspective of don’t leave your SSH keys available on open NFS mounts or open SMB shares, or even SMB shares that are shared by multiple people, because then someone can grab that private SSH key and reuse it on their environment.”

While developing improved tools — ones that won’t suffer weak default logins or any other number of security-poor shortcuts — is certainly an admirable and necessary goal, developers are left without adequate alternatives as the next generation of development platforms take shape.

In the interim, Bryan maintains that the most reliable approach is to make security a concerted part of the development cycle and not — as in some of the better development teams now (to say nothing of less diligent ones) — simply apply a supplemental security review at the end.

“It needs to be part of the process,” Bryan said. “So, as you check in code, there’s probably some sort of functionality review that happens or should happen with your code, but there should also be sort of a security review.”

Finally, Bryan advised that developers double-check not only that their development and production environments are not any more closely linked than they need to be, but also that there are no lingering points of access — like SSH keys or other login credentials — left in the development environment, in case they don’t sufficiently sever the link to the production environment.

“And then from an infrastructure perspective, again, [it’s about] cleaning up after yourself, making sure that whoever’s done the deployment has cleaned up their credentials, cleaned up their temporary files,” Bryan said. “The number of times that I come across a temp file that’s got logs or something like that that has usernames and passwords just drives me nuts.”

As hacker con season rolls along and the weather warms up, it pays to remember that a little spring cleaning — whether in your garage or your garage startup, or in a much bigger development team — goes a long way.