I'm deep into reading papers about AI, mostly with regard to secure code generation and trust in AI models. The landscape here is pretty thin though, with just a few papers commonly cited. One reason seems to be that most people still primarily focus on productivity gains with AI, and little on security or other considerations.
It's also the reason why I think AI won't have as much of a radical effect on software engineering as people think.
When no- and low-code came about, a lot of people talking about headcount reduction in software engineering simply because "most apps" could now be built in house, thanks to tools like Make or Zapier. We hear the same now that AI is used extensively by a lot of organisations. The holy grail of the argument seems to be the death of all B2B SaaS, because ostensibly those can now be crafted in house by Sales using n8n.
I think the reason we hear this so often is that people exclusively focus on productivity when talking about AI. Anyone is now able to spin up ChatGPT Work and ask it to create some app that replaces Twenty or something similarly simple1. Since ChatGPT can do it so quickly, those expensive subscriptions must not be worth it. Unfortunately, this ignores pretty much everything else needed to actually run a product successfully: Reliability, security, and ownership. The last point is also something I observed during the no- and low-code hype. Everybody wants to replace some expensive SaaS with a scruffy in house tool, but nobody wants to be called up at 3 am to fix some issue that prevents people from working on it. These things can also not be provided by AI. Code written by AI is unsafe nearly half the time (Pearce et al. 2021, 4), and AI has no concept of reliability built into the products it creates.
Given how even engineers now accept AI output wholesale due to increased productivity demands, I don't see how non-engineers (who will be expected to create and run these tools) can possibly keep up. Even though companies will surely try, I think these experiments with cutting down on SaaS costs will inevitably fail.
After all, Twenty is just a few lists and flashy cards, just like Dropbox can be replaced in an afternoon.↩︎