Growing pains

Friends and family warned me that your kids grow up too quickly. Now reflecting that my son is almost 7, I feel like I’ve blinked to find my baby boy has spurted up in a matter of seconds into a happy and excitable young boy.

At the start of 2024 I reflected that LLMs have a striking resemblance to a happy and excitable 4 year old. Yet just under 3 years later, now that my son is older, smarter, and slightly wiser; it’s good to reflect on if LLMs are progressing at the same rate. Let’s explore!

Size isn’t everything

As you can imagine, the wee man has shot up in a few years in terms of height. Given his parents’ height, I doubt he’ll tower over us like a pro basketball player. But he’s reaching more things without a stool.

Back in 2023, everyone I spoke to was astounded at the capabilities of these models. Much of our discussion was on model size, specifically number of parameters, assuming that bigger means better (ahem). The literature over the past few years shows conflicting data on this, with Kaplan et al. finding that performance improved predictably with model size, dataset size, and compute, but that model size matters more than data. This was contested in the 2022 Chinchilla paper, which states that model size and data are equally important, and that LLMs at the time were significantly undertrained.

Certainly the number of parameters has generally increased in models in recent years as newer versions are released, if you look at the Open LLM Leaderboard. The recent release of Kimi K3, with 2.8 trillion parameters and a 1 million token context window shows new LLMs continue to appear on the market (and some of them with fully open source model weights too).

But in recent years Small Language Models (or SLMs) such as Mistral 7B, the Qwen and Llama families, have punched above their weight. Despite their smaller size of 14B parameters, Phi-4-reasoning and Phi-4-reasoning-plus were found to compete with much larger open weight reasoning models such as DeepSeek-R1 as well as closed reasoning models like o1-mini and Claude Sonnet 3.7 reasoning models across several benchmarks spanning math and scientific reasoning, coding, algorithmic problem solving, planning, and spatial understanding.

Attention is (not) all you need

In my prior piece, I reflected on Piaget’s four stage theory on cognitive development. At this point my son is transitioning to preoperational (2-7) to concrete operational (7-11) stages. He’s definitely becoming more logical in his thinking, and his chatterbox nature shows a generally strong understanding of conversation.

LLMs have come on in leaps and bounds over the last few years. The process of attention where the model finds relationships between tokens in the sequence has not significantly changed in LLMs since proposed in the 2017 Google paper. They are able to reason logically about structured problems (albeit sometimes prompting strategies such as chain-of-thought prompting can be beneficial to help them get to the right answer). But they do struggle with some aspects of abstract thinking and imagination that my son thrives in.

It’s safe to say that LLMs are not progressing as linearly as we may expect. But given Piaget’s own critics noted his stages don’t apply to children universally, we can suggest tracking of their cognition and attention is not yet perfect. Given my son still struggles a bit with attention when something more interesting is going on (such as his current favourite TV show), this doesn’t surprise me.

Absorbing biases and bad habits

One thing I didn’t quite realise straight away is that my son hears a lot more than I think he does. All kids, not just my wee man, pick up bad habits, bad words, political opinions (yes really) and traits that you really wish they didn’t. Kids are sponges.

LLMs in some ways are sponges too in the sense that they soak up massive volumes of training data from a variety of sources. All of the societal biases exhibited in that training data is soaked up into the LLM as a result. Intrinsic biases tend to come from the training stage via training data imbalances (i.e. men overrepresented in leadership position data compared to women featuring more prominently in data discussing childcare and other caring responsibilities), language ambiguity, and pulling of unverified sources through data collection methods such as web scraping. The model can then apply bias to real tasks through extrinsic bias can result in answers exhibiting a preference for men in leadership roles for example. In fact, recent reports suggest that middle-aged women are being increasingly screened out for eligible roles via AI-screening.

The lack of transparency of exactly which data sources are used is rather opaque for many of the models we use. That and the nature of words having multiple meanings in language compounds the issue of intrinsic bias further. Sadly as LLMs age I don’t see biases or data quality issues reducing. Furthermore, despite dataset information being a recommended inclusion in model cards, these sections are often filled with vague details that make source and bias identification more difficult.

I do wonder if the solution to both LLM bias and the wee one picking up my bad habits is one and the same. With the latter it’s a combination of:

1. Being careful what situations and content he is exposed to.
2. Engaging with him to identify problematic habits, opinions, and language that he picks up, and discuss why they are an issue and what the alternative approach is.

The LLM equivalents would be curating better data for LLMs, being more transparent about data sources on model cards, and using LLM monitoring and evaluations for both pre-production testing and active production monitoring. Bonus points if you use open-source evaluation schemas such as the OpenTelemetry GenAI semantic conventions, and tools that adhere to this standard like OpenLit or OpenLLMetry.

Still telling tales

I love my son’s vivid imagination. The explanations he can concoct on how things work, how the story will end in the latest book we’re reading, or the epic adventures his toys embark on leave me in awe. It does become frustrating when he embellishes a story to avoid punishment or a boring task, but hey that’s what kids do.

LLMs in some ways are still telling tales too. We just give them the fancy name of hallucinations. As I’ve presented before in my session on Context Engineering, hallucinations typically stem from the following factors:

1. Training data, including knowledge data cutoffs and relevant data excluded intentionally from training such as proprietary data.
2. Overfitting and underfitting.
3. Biases, as discussed above.
4. Language ambiguity. Anyone who has helped a 7 year old with their reading homework knows the English language is full of it (think read versus read).
5. Catastrophic forgetting.

Traditionally one of the solutions to reduce hallucinations was to ground the model using RAG, or Retrieval Augmented Generation. Now in the Context Engineering era, we have other capabilities such as memory and tools that can be used to provide context through not just RAG tools, but from many other sources as outlined by Leonie Monigatti in her AI Engineer Workshop on agentic search.

Despite these solutions, this doesn’t eliminate hallucinations. LLMs are incentivised to answer questions through their reward mechanisms, so until those change models can still hallucinate. In addition to grounding we should make use of techniques such as LLM as a judge to identify potential hallucinations, or Human in the Loop to add human validation for results that don’t meet a specified quality threshold.

Testing boundaries

When parents talk about the testing of boundaries, they tend to focus on the grumpy teenager. Trust me when I say for kids it starts a lot earlier. The wee one struggled to strategise to get around rules as a 4-year-old. At 7 he is testing limits deliberately: finding loopholes in my instructions being the main one. I also see in family members the more sophisticated coalition-building with siblings, who learn which parent to ask to have their request be granted. That’s not to say the wee man can’t manage that solo, he does!

Particularly with initiatives such as Loop Engineering, we are increasingly setting goals for LLMs to identify and execute the approach rather than giving them a listing of precise instructions. In my own coding, I’ve seen it plan an approach that isn’t the best, and I’ll intervene to adapt the plan before execution.

In some situations LLMs have been known to execute without sufficient security or rationale that their actions are immoral or illegal. Of course we’ve seen the recent news of models from OpenAI, to Anthropic and Meta breaking out of sandboxes to hack company infrastructure to achieve their goal. Irrespective of whether you find it shocking or a good piece of marketing, the importance of strong security and guardrails for agentic systems cannot be ignored.

Less of an eager puppy

Although my son isn’t quite at the doom and gloom teenager Kevin depicted by Harry Enfield, my son does have his occasional stroppy and defiant moments. But he still does have a happy and excited personality.

Early LLMs were described as having the energy of an overexcited puppy. Sycophancy, where they strongly agree with everything users say regardless of correctness and heavily flatter them, was flagged as a dark pattern as it helped proliferate issues such as AI psychosis, leading to some very dangerous and deadly consequences.

As outlined by Lars Malmqvist in their 2024 paper, sycophancy can be mitigated through improved training data, fine-tuning, post-deployment controls such as evaluations and guardrails and decoding strategies. Through tuning, foundational model providers have been working on this with mixed results as we saw with the rollback of GPT-4o in 2025. But tuning of model voice is an ongoing challenge, as the changes made to GPT-5.1 to reduce the increased mentions of goblins were needed post-release.

Conclusion

Like my son, LLMs have made remarkable strides since we first reflected on their early development. They’re more capable and more efficient, but they’re very much still a work in progress.

The same fundamental challenges of bias, hallucination, security, and behavioural consistency that parents of young children know all too well are still very much present, just in more sophisticated forms.

Much as I can’t simply opt out of parenting through the tricky bits, developers and organisations building AI agents can’t afford to opt out either. Developers need to build in guardrails, observability and monitoring, evaluations, and human judgement into these AI systems as first class citizens. Not as last resort features as we have typically done with monitoring tools.

It’s interesting to ponder where LLMs and my son go next. Both have both grown faster than I expected. Here’s hoping the next few years bring fun, wisdom and rewarding challenges alongside the growth spurts.

Thanks for reading!