Happy new year!!! Yes, rather late, but I’m just starting to emerge from the shell of marking and relentless admin that forms around me at this time of year.
Anyway, it’s been about a year since I started this Substack, so I’m going to talk about what’s changed this year and attempt to get more views from review what I’ve written over this period — during which, thanks to the continuing onslaught of LLMs, computers have been fetching, decoding and executing an awful lot.
So, what was new in 2024? Well, not LLMs. Although new LLMs were released at breakneck speed, the underlying models used by them changed surprisingly little1. Sure, some notable tweaks here and there, but they’re still refinements of the GPT transformer model I dug into earlier this year in Deep Dips #3. Though something that has continued to change is their scale, with the most capable models now pushing beyond the trillion parameter mark. This scaling-up, combined with massive data sets and huge amounts of compute power, really sums up recent progress in AI.
Within this context, I think one of the key developments this year has been an increasing focus on smaller, more efficient LLMs that can be deployed within resource-limited environments, such as mobile phones. This can be seen as a response by the tech industry to the precipitous resource demands of large-scale LLMs, which have become a real concern at a time when we’re facing energy shortages and climate change. Or, if you’re more cynical, it’s about big tech pushing their models further out about our person. I talked about some of these smaller models in my post on LM Studio and open LLMs.
Something that’s been hard to miss in 2024 is the ways in which LLMs have permeated every part of the technological and socio-economic landscape. I touched on this in The Why of Generative AI. Increasingly it seems that wherever LLMs can be applied, they will be applied. Healthcare is a good example, and an area in which discussion of LLMs and AI has hit the mainstream this year, with governments seeing AI as a way of addressing limited health budgets and the strains of aging populations. But as with many things AI-related, discussion around this has been hampered by hype. As a perennial grumble-weed when it comes to ill thought-through AI/ML adoption, I addressed some of my concerns in Can AI fix the NHS?, Should we use black boxes in medicine? and “Groundbreaking AI” in medicine.
Another thing that’s changed is the ways in which we interact with LLMs. Something that’s really struck me is the take-off of in-context learning, which involves telling an LLM how to do things at inference time rather than at training time. I reviewed some of these techniques in Deep Dips #5: Prompt engineering. To me, the fact that we can now directly tell LLM things — much as we would a fellow human — rather than having to explicitly train them how to do things using optimisers and lots of data, illustrates how LLMs are becoming something more than just big neural networks.
However, it’s also clear that we retain a very nebulous understanding of how LLMs work, and I wouldn’t be doing my job if I didn’t point out the dangers of relying on them for important tasks. For instance, in Can LLMs spot machine learning pitfalls? I discussed how LLMs aren’t able to spot common errors in ML code, despite being increasingly used to generate ML code.
ML code not working has been on my mind a lot this year. As I discussed in When machine learning doesn't learn, poor ML practice remains a thorn in the side of technological progress. As my own small contribution towards addressing this, the ML pitfalls guide I’ve been working on for several years was formally published (see ML pitfalls guide updated), as was REFORMS, a guide to doing ML-based science (but not just science). In the future, I’m planning to write more posts on particular ML issues, something I started in Divvying up data, where I delved into common problems with data partitioning.
An important factor behind these ML issues has been demand for practitioners combined with a limited pool of experienced professionals. My feeling is that the increasing dominance of LLMs within the AI/ML space is making things worse, since there is now a similar mismatch between supply and demand of people with LLM expertise. That is, every company and its horse wants to integrate LLMs into their products and processes, but few people have practical real world experience of doing this2. And the relative inexperience of the practitioner population is not just a problem in industry; it’s also a problem in academic research — something I ranted about in Is peer review in machine learning dead?
And what about those who are left behind by LLMs? I touched on this in LLMs and the future of programming languages, where I speculated about the effect LLMs will have on programming language research. But it’s not just programming languages that might suffer; it’s also our understanding of how to use them. You can see this in the continuing demise of Stack Overflow, a place that programmers once met to share solutions to thorny problems. Nowadays, they just use LLMs that were trained on data from Stack Overflow. But what happens when that’s not enough — when we need to solve problems that LLMs trained on Stack Overflow don’t know how to solve?
More generally, AI/ML has had a large effect upon programmers’ choice of language. Specifically, more and more programmers are using Python, a language I shared my feelings about in Python, destroyer of worlds?. It worries me that developers are increasingly focused on a language that is slow, inefficient, and not designed to create robust systems — all pretty much the opposite of what we want in a language for AI/ML. But, alas, I can’t see this changing any time soon, given that Python has only increased in dominance in the last year. Nevertheless, there’s still plenty to learn from languages and development processes that were designed with dependability in mind. For instance, I talked about Ada and B in the Potted Programming Languages series. I also discussed the importance of types and functional languages.
But one thing deep learning has shown us is the benefit of looking to biology for inspiration, and that effective ways of doing computation don’t have to consist of lines of code. Although recent developments in deep learning have really been about scaling up simplistic models of neurons, rather than building on what we really know about how biological systems function. As an antidote to this, I delved into some of the more out-there approaches to bio-inspired and nature-inspired computing, notably slime mould computing and water-based computing. These may seem weird, but they illustrate potentially beneficial properties of natural systems that are often overlooked in conventional forms of bio-inspired computing. One of these is emergence, something I covered in Emergence in the Game of Life: a lesson for machine learning?
Computing has changed a lot in recent years thanks to the success of neural networks, yet in many ways we have only just started to change how we design computers. Even though neural processors and system-on-chip designs (both touched upon in How great Apples grew from a little Acorn) are becoming instrumental for accelerating neural networks, computers are still very much designed around conventional ways of doing computation. But as neural networks push further and further into our devices, I expect this will change. I talked about this in Is the era of the bit coming to an end? where I also discussed the influence of quantum computing. Though it’s worth remembering that all forms of computing are no more powerful than a Turing machine, justifying a precarious link to my post Turing machines and the limits of computation.
Well, that’s pretty much it, but before I go, let me reflect very briefly on a year of writing for Fetch, Decode, Execute, which gained around 100 subscribers in this time. Perhaps not a lot in the great scheme of things, but I really appreciate those of you who have stuck around and regularly view my ramblings — whilst I’m not averse to writing into the abyss, it’s nice to know that some people are listening! Whilst articles on ML/AI get the most views (yeah, looking at you Deep Dips #3: Transformers) I’m also happy to see interest in more obscure topics like Computing with Slime. Anything about programming languages receives far fewer views, though that’s not going to stop me writing about them ;-)
And neural networks themselves have changed surprisingly little since the 1980s, as I discussed in Neural networks: everything changes but you.
But on the bright side, this will provide some fruitful material for future iterations of my pitfalls guide.
Never stop. I've seen too many academic papers that read like blog posts this decade, but this is the way it should be. Well said!
Looking forward to your future posts in 2025.