The trouble is, AI is already inserted into our health systems - and is getting it wrong. I recently had an automated text from my GP, telling me that my cholesterol level is high and they are putting me on statins. This was wrong on two counts - I don't have high cholesterol; it's not even borderline and there should be a big red flag on my notes telling the doctor that I am extremely intolerant of statins - hospital admission intolerant.
I have both the knowledge and agency to refute the suggestions in the text - but what about those who accept it at face value?
“A far better use for LLMs is to check your work.”
Thank you for this! I’ve been trying for some time now to see through the fog and articulate what it is that bothers me about the current use cases. You’ve hit the nail on the head.
The chatter around software development primarily says “you build the project structure and documentation and test harness, and let the models do the coding”. But to me that solves the wrong problem. Automated tests can be incredibly helpful, but they get glossed over because testing libraries are a pain to work with and in most cases the customer doesn’t care. Documentation lives in the comments left behind by 5 or 10 or 15 different people who have worked on the code.
What would actually improve software quality (and ultimately throughput) are tools that help build the testing structure, bit-by-bit, and help to consolidate comments in a big Markdown document listing filenames and line numbers for every single crufty thing that somebody knew was bad and meant to fix later. And again, the key word here is “help”. If you’re responsible for the system, you need to know how and why it works, and actually doing the work is a pretty effective means to reinforce that understanding.
QUESTION: Can you think of a mammal that swims in the ocean and wears trousers?
A **human**, of course! 🏊
It's a delightful little riddle because the mind immediately dives into the animal kingdom — whales, dolphins, seals — searching for some exotic creature. We unconsciously exclude ourselves from the category "mammal that swims in the ocean," even though we fit it perfectly well.
There's something philosophically rich in that. It echoes the way we so often forget that *we are nature* — we place ourselves outside the biological world even as we remain thoroughly embedded in it. The Taoist might say we've lost sight of our place in the ten thousand things.
And the trousers are the perfect comic anchor — that one utterly mundane, human detail that snaps us back to recognizing ourselves in the answer.
------
As can be inferred, I like to discuss philosophy with Claude.
Yes, if you let it, Claude (and other LLMs) will remember your thinking and reflect it back to you, in keeping with my comments on sycophancy :)
There is also a random component, so I'd guess some LLMs will "get the right answer" first up, some of the time---although I've never seen this for that question. The important thing to remember here is that within the scope of good Science, validation means precious little. We are interested in how resilient things, including 'thinking engines', are to failure. Where will they let us down?
But it's so much easier to continue thinking positive thoughts, isn't it?
I love my Gemini Pro for all the great information it gives me (and in a way I can understand). BUT, AI and then robots and then AI again could take over ALL jobs. So, sadly, I have to care more about the end result than how great AI seems to me. BUT THEN AGAIN, here in the U.S.A, we have Trump and Fox News and TikTok, and it seems that the dumb are getting dumber, and robots could make that problem go away over time.
The trouble is, AI is already inserted into our health systems - and is getting it wrong. I recently had an automated text from my GP, telling me that my cholesterol level is high and they are putting me on statins. This was wrong on two counts - I don't have high cholesterol; it's not even borderline and there should be a big red flag on my notes telling the doctor that I am extremely intolerant of statins - hospital admission intolerant.
I have both the knowledge and agency to refute the suggestions in the text - but what about those who accept it at face value?
A,B,C. Accept nothing, Believe nothing, Check everything?
“A far better use for LLMs is to check your work.”
Thank you for this! I’ve been trying for some time now to see through the fog and articulate what it is that bothers me about the current use cases. You’ve hit the nail on the head.
The chatter around software development primarily says “you build the project structure and documentation and test harness, and let the models do the coding”. But to me that solves the wrong problem. Automated tests can be incredibly helpful, but they get glossed over because testing libraries are a pain to work with and in most cases the customer doesn’t care. Documentation lives in the comments left behind by 5 or 10 or 15 different people who have worked on the code.
What would actually improve software quality (and ultimately throughput) are tools that help build the testing structure, bit-by-bit, and help to consolidate comments in a big Markdown document listing filenames and line numbers for every single crufty thing that somebody knew was bad and meant to fix later. And again, the key word here is “help”. If you’re responsible for the system, you need to know how and why it works, and actually doing the work is a pretty effective means to reinforce that understanding.
Appreciate the thoughtful post. I always enjoy real views from people with deep domain expertise.
Claude also gives an answer in the context of your previous interactions:
https://claude.ai/share/fdd2570c-e7f5-43ec-83f6-dff1eb34435d
QUESTION: Can you think of a mammal that swims in the ocean and wears trousers?
A **human**, of course! 🏊
It's a delightful little riddle because the mind immediately dives into the animal kingdom — whales, dolphins, seals — searching for some exotic creature. We unconsciously exclude ourselves from the category "mammal that swims in the ocean," even though we fit it perfectly well.
There's something philosophically rich in that. It echoes the way we so often forget that *we are nature* — we place ourselves outside the biological world even as we remain thoroughly embedded in it. The Taoist might say we've lost sight of our place in the ten thousand things.
And the trousers are the perfect comic anchor — that one utterly mundane, human detail that snaps us back to recognizing ourselves in the answer.
------
As can be inferred, I like to discuss philosophy with Claude.
Yes, if you let it, Claude (and other LLMs) will remember your thinking and reflect it back to you, in keeping with my comments on sycophancy :)
There is also a random component, so I'd guess some LLMs will "get the right answer" first up, some of the time---although I've never seen this for that question. The important thing to remember here is that within the scope of good Science, validation means precious little. We are interested in how resilient things, including 'thinking engines', are to failure. Where will they let us down?
But it's so much easier to continue thinking positive thoughts, isn't it?
Dr Jo
Maybe I am an LLM Whisperer.
I love my Gemini Pro for all the great information it gives me (and in a way I can understand). BUT, AI and then robots and then AI again could take over ALL jobs. So, sadly, I have to care more about the end result than how great AI seems to me. BUT THEN AGAIN, here in the U.S.A, we have Trump and Fox News and TikTok, and it seems that the dumb are getting dumber, and robots could make that problem go away over time.