Letter to the Editor
Journal of the American Society of Information Science and Technology (forthcoming)
Sir:
In his paper entitled “Empirical evidence of self-organization?,” Peter van den Besselaar argued that the results of Leydesdorff & Heimeriks (2001) are based on unreliable statistics. Two arguments were brought against the use of discriminant analysis in our case:
ad 1. Dividing the sample
In the case of different samples which are representative of a population, one can use the eigenstructure of the first 50% for the prediction of the eigenstructure in the other 50% as a test. However, we were studying journal publications which are not representative of a population, but selected according to very specific criteria. The eigenstructures of these selections are sample-specific.
In a selected sample, the prediction can be expected to deteriorate rapidly as one extends beyond the original sample by adding new cases. The negative results reported by Van den Besselaar (2002) are therefore not surprising. His test does not work because one cannot expect any significant correlation between the eigenstructures of highly specific samples.
To pursue this line of reasoning, a better strategy might have been to proceed stepwise by iteratively recalculating the eigenstructure after each addition of a new case. (I have suggested this heuristic to Van den Besselaar in a previous exchange.)
ad 2. The simulations
Van den Besselaar claims that our results are also invalid because we disregarded his simulation results. This is simply not true: we were thoroughly familiar with his results when writing our paper. Our paper contains a reference to Van den Besselaar & Heimeriks (2000) that was published first. Furthermore, we replicated the simulations and found precisely the same results. These results were the sole ground for making the inference on p. 1266 of our paper that “(...) the weak structure in the network of words does also not significantly correlate with the geographical division.”
The second-order analysis of “self-organization” that follows after this conclusion was therefore not based on the results of the first-order discriminant analysis, as the reader can easily check. Van den Besselaar has misread and selectively quoted our paper. We repeated the argument that we had discarded the first-order results in the conclusion section (at p. 1272):
“The two types of analysis are of a different nature. Whereas no statistical significance could be retrieved inductively in the initial analysis of word patterns, these negative results did not prevent us from using word patterns in testing the second-order hypothesis. (...) (S)econd-order theorizing has a more elusive character, because one proceeds on the basis of hypothetical “what-if”-types of questions.”
We used the results of the discriminant analysis only in order to remove records that had initially been flagged as misplaced. In our opinion, this deletion was prudent because the analytical causes of the simulation results were not clear. In my opinion, the simulated data are also different from the real ones. For example, the simulation results usually did not pass the significance tests provided by SPSS (because they were based on randomness), while our results using bibliometric data did pass these tests.
Conclusion
The focus of our paper was not on data collection, but on developing a new methodology. However, our results are not invalid in terms of the statistics used for the two reasons mentioned by Van den Besselaar. His first argument is not applicable to our case, and his second argument was used by us as a reason for moving from a first-order to a second-order analysis.
References:
Leydesdorff, L., & Heimeriks, G. (2001). The Self-Organization of the European Information Society: The Case of “Biotechnology”. Journal of the American Society for Information Science & Technology 52(14), 1262-1274.
Van den Besselaar, P. (2002). Empirical evidence of self-organization? Journal of the American Society for Information Science & Technology 54(1), 87-90.
Van den Besselaar, P., & Heimeriks, G. (2000). Codification and self-organization in the European STI system. Final report to the European Commission. Amsterdam: University of Amsterdam.
Loet Leydesdorff
University of Amsterdam,
Amsterdam School of Communications Research ASCoR,
Kloveniersburgwal 48, 1012 CX Amsterdam, The Netherlands
loet@leydesdorff.net ; http://www.leydesdorff.net/