When it comes to security, you need to know what you don’t know.

In order to properly manage security vulnerabilities, enterprises need to take inventory of where, exactly, their pitfalls exist – especially as AI creates a broader area of risk. But relying only on internal “red teams,” or groups dedicated to simulating attacks on an AI model, may not be enough to identify all the hazards, said Dane Sherrets, staff innovation architect at HackerOne.

As organizations attempt to move quickly on AI adoption and development, the increasingly complex job of security can often fall on small and strapped cybersecurity teams, where flaws can fall through the cracks. These flaws can have massive domino effects. “Having more safety and security leads to more trust, which can lead to more adoption, which leads to more innovation,” said Sherrets.

Instead of relying only on internal teams to poke and prod your AI models, being “open and willing to engage with the researcher community” could reveal more about your organization’s security stumbling blocks than a single red team is capable of, said Sherrets.

• Realistically, limits on the size of an internal team, the resources of an enterprise and the diversity of members’ backgrounds can all be hindrances to finding every pitfall possible, he said.
• “To really poke and prod these models, you want to have diversity of backgrounds – and of people,” said Sherrets. “There is sort of a magic that happens when you invite people to bring in new techniques.”

While enterprises should have internal teams dedicated to identifying flaws in AI systems, that shouldn’t be the “end-all, be-all,” said Sherrets. A recent paper by Sherrets and a group of AI researchers across 24 organizations found that current flaw-reporting systems for models have major gaps, with problems often going unreported due to a lack of proper infrastructure.

Flaws in large-scale, general-purpose AI systems can pose massive risks to the consumers, developers and enterprises that use them. But building proper channels, such as standardized reporting systems, bounties and legal protections, can “incentivize researchers such as myself to spend the hours and weekends diving in deep,” he said.

“All risk management begins with an inventory,” said Sherrets. “This will allow for a bigger and better inventory.”

So where should enterprises start? If you’re building AI, the first step is making sure you have internal resources in place to accept feedback on the flaws of your systems – and personnel in place to actually do something about them, said Sherrets.

“People are going to find stuff,” Sherrets added. “So eat your vegetables first. Make sure you have teams that can own remediation and a process for acting on (flaws). Make sure you’re operationally ready.”

As the costs of data breaches continue to rise, CIOs and CISOs can no longer afford to view innovation and security as opposing forces.

IBM’s latest Data Breach report found that companies that extensively integrated security AI and automation in the prevention stages saved an average of $2.22 million in 2024. Preventive cybersecurity demands collaboration between executives and security teams, but CIO-CISO relationships aren’t always easy.

The tension between CIOs and CISOs often stems from a misalignment on how decisions should be made, said Nick Rowe, CEO of identity and access management firm Simeio.

“CIOs typically operate on a ‘possibility-driven’ mindset, constantly exploring new technologies and using a forward-thinking approach — essential for innovation but sometimes overlooking potential risks,” said Rowe.

CISOs, meanwhile, are conditioned to think in terms of “probability-driven” scenarios, constantly evaluating the likelihood and impact of security threats, Rowe said. “The difference in the way CIOs and CISOs approach the same problems, with two different styles of thinking, could contribute to a communication barrier.”

• A recent report from PriceWaterhouseCoopers found significant misalignment between cybersecurity leaders and the rest of the C-Suite: While both groups recognize the importance of measuring risks, less than half do so effectively, and only 15% measure the financial impact of these risks.
• CEOs and CISOs also have different levels of confidence regarding their organization’s cybersecurity and regulatory compliance capabilities, especially when it comes to AI.

Additionally, the role of the CIO is also changing, with many CISOs now reporting to them, rather than to boards and other executives. This makes collaboration and alignment between the two all the more vital.

CIOs and CISOs looking to bolster collaboration and strengthen their relationships can do more than just align priorities – they can integrate security into the fabric of their operations. That might include co-owning risk frameworks and responsibility, embedding security checks into development pipelines, and creating joint performance metrics reflecting shared goals, Rowe said. Such tactics can help ensure that everyone is held accountable for security.

“When CIOs and CISOs align, the benefits are endless and range from simplifying technical audits, enhancing talent retention through cross-training in high-demand areas, showcasing the best of ‘security-by-design’, and elevating cybersecurity in the boardroom and beyond,” said Rowe.

Amazon wants to give its coders a nudge in the right direction.

The company is seeking to patent “session-specific code recommendations” for editing code files. Amazon’s tech is essentially autocorrect for code, providing real-time suggestions and insights.

When a developer starts editing a code file, Amazon’s tech records the changes, using a machine learning model to analyze the patterns in the modifications. The system then suggests alternative edits or improvements to the changes the user made, which the user can then accept or reject.

“Code development tools offer developers, designers, and other users with different capabilities to improve code performance and identify errors, which may … help to overcome a developer’s lack of familiarity with a programming language,” Amazon said in the filing.

Amazon’s patent is a sign of the times: Code generation is quickly becoming one of the most common (and lucrative) use cases for the massive generative AI models that tech firms have spent the past several years developing.

• This isn’t the first time we’ve seen Amazon take an interest in coding assistants. The company introduced Amazon Q Developer in April last year after debuting the tech at Re:Invent in 2023. Microsoft, IBM and Google have launched similar tools.
• Meanwhile, Anthropic CEO Dario Amodei claimed earlier this month that AI will write 90% of code in the coming months, and AI coding startup Anysphere is in talks to raise hundreds of millions in funding at a nearly $10 billion valuation.

While tech firms are likely eyeing this as a means of achieving some ROI, AI code generators still face the same problems as any AI-powered product: hallucination, data security and accuracy issues. Amazon’s patent highlights that code assistance tech may be better put to use as a copilot to a developer, rather than as the developer itself.