We do know what is going on inside these models. The way they work at a micro and macro level is not that complicated. Scale is what produces the illusion of complexity. That complexity deceives us into perceiving that there is more going on in the system than there actually is. For such a thing to be possible, it would need to be an emergent artifact of the system, but even as an emergent artifact, it doesn't mean it is thinking. Because all large systems have emergent properties and behaviors.
There are two ways to look at this; one is that large systems approach. Large systems always have emergent properties, from traffic and the economy to the weather. Understanding that the systems do this not because of any inherent property of the individual components or interactions, but simply because of scale you can see that as we scale these system you will see emergent behaviors because it is what happens with large systems of all kinds. But again just because emergent behaviors exist doesn't really mean anything. Does the economy "think"? Does the weather "think"? Maybe, from a certain point of view they do but generally I wouldn't think most people would say they do.
The other way to look at this is from a pure data point of view. Think of the infinite monkey theorem, you know writing Shakespeare, or another more mathematical one is Pi (although personally I like thinking about Euler’s number in this context). The idea here is that any irrational number (or an infinite number of monkeys banging on typewriters) theoretically contains all knowledge encoded in it but not just just all knowledge all knowledge about knowledge in every organization model possible. Because the numbers are infinite and do not repeat they contain somewhere in them all conversations that have ever have taken place and ever will take place in every language ever known or yet to be created. Now if this data exists you just have to find it. The thought experiment is this, you could have an entirely intelligent conversation with Pi or Euler's number if you started at the correct digit. It would appear intelligent, it would appear to think. But it would be none of those things. It's just data. And that is the analog here, it's an over simplified way to illustrate that even basic data can appear intelligent regardless of computation.
As far as studies go, in laymen's terms "inconclusive" results would seem to leave the possibility open but scientifically speaking "inconclusive" results equates to "No." Now that is not a hard No it what would be soft No. Scientists are precise in their communication and they will almost never say something is not possible because proving a negative is almost always logically impossible. (it's like the black swan problem) Basically when you hear words like inconclusive from the scientific community you should just think "No." because that is basically what they are saying. But scientists will always hedge, so they will almost never categorically dismiss a negative proposition. More importantly, when consuming any media regarding AI practitioners consider their biases and motivations these people are not evil but they have vested interests.
I brought up the point of the efficiency of the human brain not as a goal for current AI research to obtain but to point out that obviousness of the fact that the way we are going about trying to replicate human-level intelligence is obviously completely wrong. We are currently brute forcing a poor and limited imitation of the human mind by using billions of times the amount of energy required to run an actual human mind. If that is what is required to get just to where we are now clearly we are doing something wrong and are on the wrong path. How far can we go down this path? It is hard to say, maybe we will go far. However because we've missed something, and we are on the wrong path and the current approach is sucking all of the air (and money) out of the room we are not going to get to true human-level intelligence anytime soon. We will definitely not get to it with the current approach that everyone is on. We will have some great tools that came out of this process but unless the fundamental approach is changed we will not reach human-level GAI. That change will most likely require yet another AI winter.
