Test prepping discussion

From reddit http://www.reddit.com/r/Khan/comments/1znhcx/khan_academy_gets_rare_partnership_to_close/

Me:

Test prepping does not work very well, so it’s a minor issue. SAT and ACT tests are mainly tests of g and one cannot train g.

Him:

That’s a common misconception, but they’re not general intelligence tests. SAT and ACT test very specific material that can be studied, so test prep actually makes a huge difference in scores.

Me:

No. Try these: http://pss.sagepub.com/content/15/6/373.short http://infoproc.blogspot.dk/2012/02/test-preparation-and-sat-scores.html

Him:

I’m a little too tired to look up studies and pick apart the methodology, so I’m just going to make a couple general points. First and foremost, why are the writers of the sat revising the test to make it less susceptible to test preparation (by their own admission) if it’s not affected by test preparation? Why change it at all if it’s an accurate test of general intelligence? Why is family income so strongly correlated to sat scores if the test score can’t be affected by other factors? Why would so many elite colleges be turning sat optional if the score actually represented human intelligence? Are native English speakers naturally smarter than not native speakers since native speakers have higher average scores?

Me:

I’m a little too tired to look up studies and pick apart the methodology, so I’m just going to make a couple general points. First and foremost, why are the writers of the sat revising the test to make it less susceptible to test preparation (by their own admission) if it’s not affected by test preparation? Why change it at all if it’s an accurate test of general intelligence?

Maybe because people like you think like this.

Why is family income so strongly correlated to sat scores if the test score can’t be affected by other factors?

Because high g parents have high g children and high g parents earn more money. Common cause.

You should drop the straw man. I did not say that it was not affected by other causes.

Why would so many elite colleges be turning sat optional if the score actually represented human intelligence?

For political reasons? To easier meet racial quotas?

Are native English speakers naturally smarter than not native speakers since native speakers have higher average scores?

If you use a verbal test on non-natives then you get biased results. That’s why one uses non-verbal tests instead.

Here’s a study for ACT. http://www.sciencedirect.com/science/article/pii/S0160289607000487

But really, that SAT/ACT are mostly measures of g has been known for decades.

There’s a replication here for the 2004 study: http://www.sciencedirect.com/science/article/pii/S0191886906000869

Him:

Maybe because people like you think like this.

Not sure what you mean. People with relevant expertise?

You should drop the straw man. I did not say that it was not affected by other causes

It’s not a straw man, I’m challenging an assumption necessary to your argument. You predicated your claim that sat scores can’t be changed by prep on the (dubious) premise that the sat is a test of general intelligence, thereby assuming that tests of g can’t be improved by prep. I’m demonstrating that the scores can be affected by other factors, and positing that a test that can be affected by myriad other factors can be influenced by targeted practice.

All of this boils down to the simple truth that I know for a fact that students’ scores improve with practice. It doesn’t matter how many tangentially related studies you cite, I’ve seen the unequivocal reality hands on. I’m working with 3 students right now. One has gone up 200 points on the sat, one has gone up 300, and the last has gone up 4 points on the act.

I really gotta ask, have you ever taken either test? What did you get? It’s hard for me to imagine that someone who knows the question types actually thinks the sat tests human intelligence. (I won’t go so far as to say that about the act, but it’s still on material that can be practiced)

Me:

It’s not a straw man, I’m challenging an assumption necessary to your argument. You predicated your claim that sat scores can’t be changed by prep on the (dubious) premise that the sat is a test of general intelligence, thereby assuming that tests of g can’t be improved by prep. I’m demonstrating that the scores can be affected by other factors, and positing that a test that can be affected by myriad other factors can be influenced by targeted practice.

It is a strawman. I did not claim that one can’t train SAT scores. I specifically said one could, but not much.

One can improve IQ scores (manifest variable) but not g (latent variable). See: http://www.sciencedirect.com/science/article/pii/S0160289606000778

All of this boils down to the simple truth that I know for a fact that students’ scores improve with practice. It doesn’t matter how many tangentially related studies you cite, I’ve seen the unequivocal reality hands on. I’m working with 3 students right now. One has gone up 200 points on the sat, one has gone up 300, and the last has gone up 4 points on the act.

http://en.wikipedia.org/wiki/Anecdotal_evidence

Him:

You’re the only one here talking about g. This is an article about the SAT, not g.

And are you really claiming that the universal consensus of everyone exposed to test prep is simply anecdotal evidence that you, a dilettante, can see right through? Again, ETS, the writers of the SAT, have just stated that it’s too influenced by test preparation. You know more about it than they do?

Me:

You’re the only one here talking about g. This is an article about the SAT, not g.

You’ve also been talking about it above.

And are you really claiming that the universal consensus of everyone exposed to test prep is simply anecdotal evidence that you, a dilettante, can see right through? Again, ETS, the writers of the SAT, have just stated that it’s too influenced by test preparation. You know more about it than they do?

No evidence for any universal consensus presented. One can see that this isn’t so on Wikipedia as well. https://en.wikipedia.org/wiki/SAT#Preparations

I already stated one can increase SAT scores by training, but not much.

Following the systematic review mentioned indirectly on Wikipedia: http://onlinelibrary.wiley.com/doi/10.1111/j.1468-2397.2011.00812.x/abstract

The mean gains on V and M were 24 and 33 points. Compare this to the published standard deviations of the subtests of around 114-118, you can see that this training did not do much. About .25 SD increase. Equivalent of 3.75 IQ points (SD15).

http://media.collegeboard.com/digitalServices/pdf/research/SAT-Percentile-Ranks-2013.pdf

Him:

The Wiley link above is the most recent article you’ve mentioned, and it still implies that data quality in the field is poor. And while I would dispute that the figures they give would be accurate for my work, they still called the gains from test preparation significant. From the article:

“As long as coaching remains inaccessible to some students, we urge universities to reconsider the weight given to SAT scores in the undergraduate admissions process. We challenge the designers of the SAT to redesign the examination to eliminate the possibility of score gains from coaching. Finally, we call for researchers to increase the production of high-quality data in this field to ensure accurate estimates of coaching’s effects are made available to all.”

And I gotta come back to this. Have you ever even seen an SAT?

At this point it seems like he had given up trying to argue, and merely wanted to talk about other stuff.

One can recap it in terms of references given:

Me – 9

Him – 0