I'm deep into reading papers about AI, mostly with regard to secure code generation and trust in AI models. Compared with the enormous amount of dfiscussion around AI productivity, the security literature still feels relatively small. One reason seems to be that most people still primarily focus on productivity gains with AI, and little on security or other considerations.
This focus on productivity also shapes how people predict AI will change software engineering. We talk extensively about how much cheaper software is to build, but much less about what happens after it has been built.
When no- and low-code came about, a lot of people talked about reducing software engineering headcount simply because "most apps" could now be built in house, thanks to tools like Make or Zapier. We hear the same now that AI is used extensively by a lot of organisations. Taken to its extreme, the argument predicts the death of B2B SaaS: why pay for software if Sales can build what it needs in n8n?
I think the reason we hear this so often is that people exclusively focus on productivity when talking about AI. Anyone is now able to spin up ChatGPT Work and ask it to create some app that replaces Twenty or something similarly simple1. Since ChatGPT can do it so quickly, those expensive subscriptions must not be worth it. Unfortunately, this ignores pretty much everything else needed to actually run a product successfully: reliability, security, and ownership. The last point is also something I observed during the no- and low-code hype. Everybody wants to replace some expensive SaaS with a scruffy in house tool. Far fewer people want to own that tool when it stops working at 3 am. AI can help build and maintain these systems, but it cannot assume responsibility for them.
Security is a good example. Early research into Github Copilot found that roughly 44% of generated programs in security-sensitive test scenarios were vulnerable (Pearce et al. 2021, 4), though results have improved with newer AI models. Now, code written by AI is roughly as insecure as code written by humans (Belozerov et al. 2026, p .4). AI has reduced the cost of producing code, but it hasn't removed the need for someone to be accountable for the resulting system.
Giving non-engineers the ability to build internal software doesn't remove the operational burden. It transfers that burden from a SaaS vendor to the company using the tool. Some of these experiments will work, particularly for small, low-risk internal tools. But as the importance of the system increases, the cost shifts from building the software to owning it.
After all, Twenty is just a few lists and flashy cards, just like Dropbox can be replaced in an afternoon.↩︎