How I learned to stop worrying and love SecOps
Note: this post is about defensive security that helps developers build secure applications instead of offensive security.
Much ink has been spilt to describe and praise the recent DevOps (developer + operations) phenomenon. The development community rejoiced as we tore down the walls between dev and ops teams. We cried tears of joy, and product-owners high-fived each other as DevOps shortened the software development lifecycle and provided continuous delivery with high software quality.
However, in this brouhaha, we seem to have forgotten (or ignored) an element of Ops that is a crucial part of software development - Security. Many corporations (there are of-course exceptions) seem to have delegated security to a secondary status that is viewed more as a blocker than an enabler. There are good reasons for this:
Developers are still not actively taught about secure coding practices when learning a programming language or framework. They are expected to learn independently or by following their organisation’s "coding standards". It's akin to teaching them how to drive a car but not how to put a seatbelt on or change lanes safely.
The security team/culture either doesn't want to evolve or is not empowered to change to match the cadence of the modern continuous delivery process. The classical approach - doing pen testing after the fact or doing application security review days before go-live - doesn't scale and slows the team down. If a process introduces blocking, it will be ignored, not embraced.
Security is still thought of as an after-the-fact task. It is not as crucial as shipping a functional product. This attitude also applies to security testing. Many startups don't have dedicated security engineer(s) on their team until they have x number of members in the organisation, or y number of customers or z millions in revenue.
Over-reliance on a particular framework to do the work for you. Many teams believe that if they use a framework that promises baked-in security, they're covered and don't have to worry about application security. Unfortunately, there is no such thing as a secure language or app framework.
Doing security right is hard. It's like trying to build a parachute after you've jumped and before you hit the ground. In teams where devs push code to production 100 times a day - how do you ensure what's being pushed is secure and follows best practices without being a blocker?
Given these challenges, is it possible to create a culture where security is an enabler and not a blocker? The short answer is yes, but it requires a massive shift in attitude and incentives linked to delivery (see the long answer below).
The solution lies in extending the principles of DevOps (i.e. remove silos and organisational blockers, focus on good communication and collaboration) by incorporating security since it faces the same challenges as Ops.
An oft-repeated cliche in the security industry states - Security is everyone's responsibility. It's like saying He who controls the spice controls the universe1. It's evident and meaningless. Unless you make cultural and fundamental process changes and provide the right incentives to promote security, it will NOT be everybody's responsibility.
I don't claim to have all the answers, but here are a few things I've learned that will help you set up a DevSecOps process that might help:
Start with culture
Security is as much a people problem as it's a technology problem. Historically, we didn't discuss the need for positive security culture. The approach to security was more technical and less human - we tried and failed to solve the security problem by building or buying tools to detect, fix and deter security issues.
Hire a dedicated security team. Depending on your requirements and budget, you can have engineers, pen-testers, security architects. Pro-tip: make sure the team is diverse.
Embed designated security engineers and architects in your product/project team squads. This approach is a simple extension of embedding Ops engineers as part of the team. The security engineer is involved in sprint planning, design session, code reviews (where possible), building personal relationships, educating the peers on security best practices, and creating graphs/stats to track security metrics. By making security part of the development lifecycle, you.
The security engineer should act as an enabler (they should be incentivised based on this criteria) and help build tools or use off the shelf tools that will allow them to keep up with the development cadence.
Remove the barrier to interact with the OpSec team - make them accessible and visible, a.k.a don't put them in the basement.
Collect data and metrics to understand the ROI of your security investment.
Number of tickets/user-stories that impact access control, handling sensitive data
The number of defects related to security. Use these to compare which feature set of the platform has more issues and do a root cause analysis to understand if there is a fundamental problem with the development methodology for the application framework, e.g. android app has more bugs than the iOS app.
Bug bounties (from external users or security researchers)
The number of known breaches per day/week/quarter. It will make the need to have a more secure approach explicit
Real-time monitoring of attacks (e.g. DDoS)
Evolve the rules and process as the company and the product evolves. Ensure that your best practice reflects the latest and the best approach to securing applications
The data serves two purposes -
It helps you learn from your mistakes and evolve your security posture.
It makes security incidents explicit so everybody can see why we need to be proactive about security.
Management / Exec Support
The culture change to be more security-minded needs to be driven top-down. Hire a CISO (Chief Information Security Officer) if you can. Don't (under any circumstances) give the CISO responsibilities to someone responsible for technical delivery to avoid conflict of interests. Keep the two roles separate and incentivise the CISO to be an enabler for the tech/delivery team.
Blameless postmortem: When a security incident does happen, ensure that the postmortem is blameless. These postmortems aim to identify and understand the root cause instead of finding a scapegoat. The learnings from the postmortems must flow back to the implementation team to prevent similar issues in the future.
The security industry still has a long way to go to be seen as a crucial part of the development lifecycle. There is reason to be optimistic, however. The DevOps approach is gaining traction, and it will be the default mode of building products and applications soon. It paves the way for treating security the same as Ops, and inserting it as part of the delivery will be easier. The suggestions in this post are one way of doing this, more effective and nuanced approaches will evolve, and it will benefit everybody - developers, end-users, and the organisation's bottom line.