What Is Your Students’ Average Score Increase?

Who knows? 🙂

I don’t mean to be flip. Actually, there’s a lot to be said on this topic, because I would argue that nobody knows for sure what the average results are for any test-prep company.

On top of that, even if we could know the average score increases, the number wouldn’t really be useful anyway.

Let me explain:

Reasons Why It’s Impossible To Track Score Increases Accurately

In order to track score increases, a test-prep company would have to have access to its students’ score reports. If the company doesn’t have access to students’ score reports, it has to be relying on students to report their scores instead. Relying on reported scores would be a bad idea from a scientific point of view, because (a) people could lie, either on purpose or by accident, and (b) not everyone will bother to report a score, meaning that the reported scores will be likely to come from customers who are more passionate about the company, which will mean that the reported scores might be skewed higher than they should be. So it’s not really possible to get accurate data without having direct access to a student’s official score report, which isn’t possible.

On top of that, there’s the fact that many students use more than one test preparation method as they get ready for a test, and some students don’t pay full attention to what they’re taught in a particular course. If a student signs up for a course with Kaplan and then decides halfway through that he’d rather try my approach instead, should his results be counted as an indicator of the validity of Kaplan’s ideas, or of mine? So even if you could get the scoring data for a particular student, it would be impossible to say for sure that the student was only influenced by one company, and that the student’s results (whether positive or negative) were a true reflection of the quality of the company’s instruction.

Why Knowing A Student’s Score Increase Might Be Misleading

When we talk about standardized test scores, it’s important to remember that all standardized tests have a “perfect” score that can’t be exceeded, and that all tests measure multiple skills. This means that score improvements of the same size might be easier or harder to get at different points in the scoring scale.

To put this in concrete terms, it’s typically much easier to help a student go from a 130 on the LSAT to a 140 than it is to help the student go from a 170 to a 180. The improvment in both cases is still 10 points. But a student who’s scoring 130 needs a lot of help and can choose from a variety of skills to improve on, and he can afford to make a large number of mistakes on any given day while still improving his score. A student who’s scoring a 170 has probably already made all the easy adjustments he can think of, and he has to make almost no mistakes at all if he wants to improve to a 180.

So new points generally become harder to get as we move towards the upper limit of the scoring range. If a company were to report that its average LSAT client improved by 15 points, would that mean that it tended to work only with people who started out being pretty bad at the LSAT? If a company said its average increase was 5 points, would that mean that the techniques were useless or that most of its clients were already in the 170’s before using the course? There would be no way to tell.

***

So, to put it concisely, I don’t try to track student performance because I’m a big believer in science and numbers, and I know that

(a) there’s no reliable way to get a student’s score,

(b) there’s no way to be sure that a student’s score had an outside influence, even if I could find out that score, and

(c) the size of an increase or decrease is meaningless without knowing where in the scoring range that increase or decrease occurred.

I would strongly advise you to be suspicious of any company or tutor that tries to impress you with an “average score” increase.

***

So what’s a test-prep tutor to do, if he can’t persuade potential customers with a meaningful indication of an average score increase? Sounds like a big problem, right?

Well, not really. My solution is this: I offer free videos and articles on my web sites that give you an idea of how I think and how I teach. You can even watch me work on some real practice questions to get a taste for the course. So you can already have some idea of whether you like the kinds of things I do. Then I back everything up with the strongest guarantee I’ve ever heard of in this field: If you try my course out and don’t like my approach, all you have to is let me know about it within 60 days and I give you your money back. No fine print :).

So while I can’t offer any meaningful data on the average user experience (and neither can any other test prep company, in my opinion), I do everything I can to give you a good idea of whether you’ll like my approach in advance by showing you some free videos, and I take your risk away by offering the strongest guarantee in test prep. (By the way, I can only do those things because I know from customer feedback that most of my students are very happy with my approach–I just prefer to be honest about the fact that I don’t have any meaningful way to quantify that happiness.)

***

One more little sidenote–you might be asking yourself, “Can’t he just tell me about the specific score increases of a few specific kids he’s worked with who did really well?” The answer, strangely, is no, I can’t. Which sounds crazy, but it’s true–the FTC recently changed the rules to make it illegal for a company to talk about quantifying a specific customer’s result without giving information about the average customer’s result.

http://www.ftc.gov/os/2009/10/091005revisedendorsementguides.pdf

(They supposedly did this to protect consumers from claims made by the people who advertise weight loss supplements and get-rich-quick real estate programs, but the rules apply to everyone.) Since I have no way of knowing what my average student’s score increase is, I can’t tell you about specific students of mine, either. It’s a shame, because I would really, really like to–but that’s just another reason why I offer my blog and my guarantee. You can get a ‘taste’ of how I do things by looking around here, and if you decide to sign up, you have absolutely no risk in case you decide it isn’t for you.