All speakers
Mr. Vikram Pandya
Eminent Speaker

Mr. Vikram Pandya

Director, AI Initiatives, S P Jain Global

Day 1 · Leadership Discourse 2
When the measure becomes a target, it ceases to be a good measure.
Goodhart's Law — the line that frames the entire session
Post-event

Session recap

The data you trust is often the data lying to you. The fix is the question, not the dashboard.

Vikram Pandya opened Leadership Discourse 2 with the disarming image of a genie. Ask the genie for three wishes carelessly, and you get exactly what you said — and never what you meant. Data and AI work the same way. The principal who walks into a meeting with a dashboard and walks out with a wrong answer is almost never the victim of bad data. They are the victim of a wrong question, pointed at the wrong dataset, interpreted through the wrong lens. Most of the time, he said, schools are getting more noise than signal — and the work of leadership is to flip that ratio.

The single most important idea of the session was Goodhart's Law: when the measure becomes a target, it ceases to be a good measure. The example landed in seconds. Tell teachers their performance will be judged on a 10-out-of-10 feedback score, and within a term you will see fun-activity-driven classrooms and exam questions designed to be easy. The number goes up; the learning goes down. Sales teams under sales-only targets mis-sell. Universities under ranking targets game rankings. The measure was meant to describe excellence; the moment it becomes the goal, it actively erodes it.

Before getting to the wrong-question / right-question reframe, Vikram established a clean three-part diagnostic for the data itself. Incomplete data — fields not filled, parents who didn't respond, attendance records with proxies. Inaccurate data — totals that don't tally, IDs that don't match, withdrawal reasons that aren't the real reasons. Data that exists but isn't ready for analysis — scattered across systems, in different formats, never assembled in one place to be queried. Tables worked through their own examples, and the room watched the same pattern emerge across schools: the first two are process-and-control problems; the third is an architecture problem. None of them are an "AI" problem.

If you are not clear, and you are not asking the right questions — you will not get the right response. Data and AI together are like a genie. They will give you exactly what you asked. Never what you needed.

He then introduced the five statistical ghosts that haunt school data even when the data itself is clean. Simpson's paradox — aggregates that reverse direction when broken into subgroups (an 87% pass rate that hides a 61% section). Survivorship bias — looking at bullet holes on the planes that came back, not on the ones that didn't; celebrating toppers while never analysing the students who didn't make it. Correlation is not causation — students who attended more classes scored higher, but did the attendance cause the marks, or did the discipline that drove both? Wrong denominator — comparing percentages from different baselines and drawing dramatic conclusions from noise. Goodhart's Law — the one that opened the session, because it sits underneath all the others.

The right-question / wrong-question reframes were the most actionable section. "How many applications did we get this year?" is the wrong question (top-of-funnel, vanity). "At what point in the funnel are we losing prospective parents, and how fast are we responding?" is the right one. "What is our overall pass percentage?" is the wrong question (aggregate). "Are we improving outcomes for the bottom quartile, or only protecting the top?" is the right one. "Who are our fee defaulters?" is the wrong question (people). "Is this a willingness-to-pay problem or a system-design problem?" is the right one. The pattern is consistent: every wrong question names what is; every right question asks why it is, and for whom.

Vikram closed with a design lesson hiding inside a feedback example. If you ask users for a 1-to-5 star rating, you will get a skewed-negative signal — unhappy people review, happy people leave. The fix is not to abandon the metric, it is to design friction into the negative path: require detail, require context, let AI parse whether the negative is real. Schools, he said, can apply the same principle to admissions, parent surveys, teacher feedback and exit interviews. Build the system so the data you collect is the data you can actually trust. Then ask better questions of it. That is the work.

Editorial summary compiled by the LDP team — not a verbatim transcript. Spotted an inaccuracy? Let us know.

Key takeaways

  1. 1

    Goodhart's Law, in one line

    The moment a measure becomes a target, people optimise the measure instead of the thing it was tracking. Audit every metric your school treats as a target — feedback scores, pass percentages, admission funnels — and ask what it is now distorting.

  2. 2

    The three data problems

    Before you ask anything, classify the data: incomplete, inaccurate, or not-ready-for-analysis. The first two are process problems. The third is an architecture problem. None of them are solved by adding more AI on top of bad data.

  3. 3

    The five data ghosts

    Simpson's paradox, survivorship bias, correlation-vs-causation, wrong denominator, Goodhart's Law. Learn the names. The moment you can name the ghost, you stop being haunted by it.

  4. 4

    Wrong question → right question

    Every wrong question names what is. Every right question asks why it is, and for whom. Build the habit at the leadership table: when someone shares a number, ask "what would the better question be?"

  5. 5

    Drill down past the aggregate

    A school-wide 87% pass rate that hides a section at 61% is not good news — it is two stories pretending to be one. Disaggregate every important metric by section, by subgroup, by ability quartile.

  6. 6

    Design friction into your feedback

    Negative reviewers will always be louder than happy ones. Build systems where giving a negative rating requires detail, context, and AI-assisted parsing — so what you collect is what you can actually trust.

What he spoke on

  • Why "noise vs signal" is the real data problem in schools

    Most schools collect more than they can interpret. Signal is the small set of metrics tied to a decision; everything else is noise that crowds the room and tires the leader.

  • The two wishes problem with AI

    Like a genie, AI gives you what you literally asked. The leader's job is to translate the messy real question into a precise prompt — and to know when to override the answer the system gives back.

  • Goodhart in the classroom

    The day faculty feedback scores became a promotion criterion was the day faculty optimised for friendliness over rigour. The same pattern shows up in pass percentages, admission counts and teacher-attendance metrics.

  • Survivorship bias in every annual report

    Every school publishes its toppers. Almost none publish what happened to the students who didn't make it. The story you don't tell is the data you don't see.

  • Correlation is not causation, and gut feeling is not causation either

    When teachers blame the previous batch, the syllabus, or the calendar for a poor result, the leader's job is to ask: prove it scientifically. Opinions cause arguments. Facts cause answers.

  • Friction is a feature, not a bug

    A good feedback system makes it slightly harder to leave a one-star review than a five-star one. Not to suppress complaints — to ensure the complaints you receive carry the context needed to act on them.

Q&A captured

Q. How should we identify incomplete vs inaccurate vs not-ready data in a typical school?

Incomplete is "fields are blank" — birth certificates not submitted, parent feedback responses missing. Inaccurate is "fields are filled with the wrong values" — totals that don't tally, IDs that don't match. Not-ready is "the data exists but lives in five different systems and was never assembled" — admission data in one ERP, transport in another, examination in a third. Treat them as three separate problems with three separate fixes.

Q. How do we avoid the survivorship-bias trap when reviewing student council selections?

The trap is selecting only on the criteria you already wrote down. The fix is to look backwards as well: identify past council members who turned out excellent, and ask which of their qualities weren't on your selection criteria. Add those qualities to the criteria. It's a scientific way to expand the list — and it forces you to study the data you would otherwise have thrown away.

Q. When teachers blame results on "this batch was weaker than the last," is that data or opinion?

Opinion — until you prove it. Compare year-on-year inputs scientifically: entry scores, attendance, participation in activities, teacher continuity, syllabus completion. If the inputs really were weaker, the data will show it. If not, the explanation is somewhere else and the conversation needs to go there.

Q. How should we treat the 4% of parents who don't pay fees in a given month?

The 96% who paid is the wrong story. Ask: of the 4% who didn't, how many never pay, how many pay late every month, how many usually pay but missed this once? Three different patterns, three different interventions — collapsed under one number.

Q. Can AI help with classroom-observation data?

Yes — AI can analyse a recorded lesson against the lesson plan and tell you what was covered. But "covered" is the wrong question. The right question is "was it effective?" — were students applying, questioning, associating? AI can score both, but only if you ask both. Don't let the easy metric crowd out the one that matters.

View the pre-event page (archived)— bio, predicted topics and pre-reads as shown before the session.