In a recent virtual roundtable among some of the industry’s top leaders, Andrej Karpathy did not hold back. He emphasized the need for caution in deploying artificial intelligence (AI). He asked developers and industry leaders to “keep AI on a leash.” He focused on the dangers of high-stakes, real-world use of large language models (LLMs) with no reliable human accountability in place. This call for vigilance comes as AI technology continues to advance rapidly, raising concerns about its implications for software development and employment.
Recently, Andrej Karpathy lamented that LLMs are fundamentally broken and will return garbage results without supervision. He explained that these systems frequently misunderstand the intent of user prompts, leading to unexpected or even harmful results. Yet this unpredictability is the very reason we need human engineers at the helm of the process. They can respond to unforeseen consequences, set standards of AI behavior, and address nuanced issues that AI is not yet capable of comprehending.
The Role of Human Engineers
Kent Beck, one of the original co-authors of the Agile Manifesto, offered this fascinating analogy. He compared AI agents to genies granting wishes, but warned that these genies don’t understand the nuance behind those wishes. Interestingly, this comparison highlights the importance of human intervention in steering AI development. Bob McGrew, a former research lead at OpenAI, has very high conviction that human engineers will always be important. They can be critically important when it comes to making sure AI works and works safely.
Software development and especially open source is in many ways the flagship example of this trend. And as recently as this week, Sundar Pichai announced that over 30% of Alphabet’s recently generated code is AI. That’s a significant increase from the 25% reported last year. While this stat drives home how deeply AI is becoming ingrained in coding across the industry, it begs fundamental questions about the future of developer occupations. We think Pichai is missing the point when he insists that AI is intended to create jobs, not destroy them. It’s evident that leveraging AI in coding workflows is changing the game for developers.
Regulation and Oversight
Karpathy expressed that deploying AI at coding scale requires significant guardrails. Developers are turning to AI tools to create thousands of lines of code in seconds. This quick pace of development leads to new worries around the potential for harmful use or misinterpretation. We agree that the unprecedented speed of AI development necessitates proactive oversight so governments and industries can reduce harmful effects from AI deployment.
We know that AI is a powerful force for establishing new levels of productivity and efficiency. As the field rapidly develops, let’s not forget that human expertise will remain irreplaceable. For developers, embracing this new paradigm will require learning to work alongside AI, and not be supplanted by it. Technology is always changing. If we’re to truly unlock the potential of AI, we need to put humans at the center and keep the balance.
