Demanding Explainable AI, Recommending Black Boxes

There’s a demand that shows up in nearly every AI strategy and that matters to me. AI systems need to be more traceable. Companies should understand how their models reach decisions, because transparency is supposedly critical for trust. For me that means: without explainability, no responsible use.

Explainable and transparent AI is, in my view, an important demand.

The concrete tools everyone recommends are ChatGPT, Claude, DALL-E, Midjourney, and it feels like there are more every day. Tools built on large language models whose inner workings even their developers don’t fully understand. Nobody at OpenAI can explain to you in detail why ChatGPT gives a particular answer to a particular question. Not roughly, but in detail. And that’s not acceptable. A product is being created that is so unpredictable nobody understands it. That’s not OK. But the models are too complex. They have billions of parameters whose interactions are beyond any human comprehension.

These are black boxes, by definition, not as an accusation but as a technical fact. A prompt goes in and the output is instantly there. Change even one insignificant word in the prompt and the result is completely different. Something is happening that no one on earth can fully explain.

But these people promote their tools on one side as the perfect all-purpose weapon for everyone and everything and explain nothing. Absolutely nothing. My first thought was whether this is ignorance or intent. Of course it’s intent, because it’s ignorance. You have to deliberately obscure something because you don’t understand it yourself. The facade is a trade secret and we as outsiders have long gotten used to that. We use tools every day that we don’t begin to understand. Every one of us. I don’t understand in detail how my phone works and I use it anyway. But my phone doesn’t make substantial decisions.

The difference between a tool you don’t understand and a tool that makes decisions you don’t understand is fundamental. With the first, you trust the result because you can check it. The text is good or bad. The image fits or it doesn’t. You are the corrective and you have enough judgment. With the second, you trust the result because you have no other choice. The AI says: this applicant is qualified and this one isn’t. Based on what? Of course there are sophisticated algorithms behind it and the answer isn’t bad per se, but who actually checks the result?

The whole industry moves between both worlds without explaining the difference. It recommends ChatGPT for writing and calls that productivity. It recommends AI-based analytics for business decisions and calls that transformation. But nobody seriously demands explainability, because it’s a trade secret.

What bothers me most about this is how casual it is. Hardly anyone engages with the black box discussion, even though it’s urgently needed. The question is not whether AI is useful, but whether you’re aware that you’re making decisions on foundations you can’t trace.