Weighting
Description of how a study is weighted on Jobeffekter.dk
Not all studies have the same quality and relevance, and consequently, not all studies should be given the same weight when overall conclusions are drawn across studies. Therefore, the studies are weighted such that results from studies of low quality or low relevance are given less weight than the results of studies of high quality or high relevance. Weighting is based on the following criteria:
- The researcher’s quality assessment
- Analytical method
- Geography
- Published in a scientific journal/relevance of scientific journal
The criteria for weighting and the size of the weights are established in collaboration with the independent researchers contributing to developing Jobeffekter.dk. The maximum weight achievable by a study is 1, and this is given to a randomized controlled trial, published in one of the best scientific journal, and with the highest possible quality assessment from the researcher uploading the study.
The weighting forms part of the calculation of the knowledge contribution used to assess the overall stock of knowledge and level of evidence for the overall conclusion about effects (the job effect).
In the following, the four weights are reviewed.
Quality assessment
When uploading a study, the researchers assess the quality of the study. The quality assessment takes into account the data quality and method of the study. In this connection, it is important that the study is based on a sufficient number of observations, that other effects that might potentially influence the outcome are taken into account, and that the study includes an assessment of the robustness of its results.
The researchers assess the study according to the following scale:
|
Quality assessment |
Weights |
|---|---|
|
Good |
1 |
|
Satisfactory |
0.5 |
|
Not Satisfactory |
0.25 |
The weighting implies that the estimated effects of a study assessed as “not satisfactory” are given less weight than the estimated effects of a study assessed as “satisfactory”.
Analytical method
When uploading a study, the researcher weights the analytical method used in the study according to the following scale:
|
Analytical method |
Weights |
|---|---|
|
Controlled trial |
1 |
|
Econometric/statistical effect study |
0.8 |
The weighting means that estimated effects of randomised controlled trials are weighted higher than analyses based on econometric/statistical effect studies. The reason being that the use of controlled trials ensures that individuals belonging to the target group for the intervention are randomly assigned to groups either receiving the intervention or functioning as a control group. This guarantees that the estimated effects can be ascribed solely to the interventions, and not to any unobserved differences between the intervention group and the control group.
Geography
When uploading a study, the researcher registers the country from which the study originated. However, a study is only uploaded if it originated in the OECD. This decision is based on an assessment of the labour market structures that have the greatest resemblance to Danish structures, and which are thus likely to be most relevant for the Danish labour market. Based on the country registration, the study is weighted according to the following scale:
Note that weighting is deactivated in the English version of Jobeffekter.dk when the job effect is calculated.
|
Analytical method |
Weights |
|---|---|
|
Denmark |
1 |
|
Northern Europe |
0.75 |
|
OECD (excluding Northern Europe) |
0.5 |
The labour market structure on the Northern European labour markets is more similar to that of the Danish labour market with respect to social benefits, active intervention, flexibility and structure of earnings than the labour market structure in the other OECD countries. Thus, results from Northern Europe are likely to be more easily transferable to the Danish labour market, and consequently, a study from Northern Europe is given more weight than a study from other OECD countries.
Published in a scientific journal
When uploading a study, the researcher registers whether the study has been published in a scientific journal. If the study isn't published in a scientific journal it will get the weight 0,5. Is the study published in a scientific journal this will be registered based on the journal, being ranked by international standards and receive a weight from 1 to 0.6:
|
Publication of results |
Weights |
|---|---|
|
Results published in a scientific journal (ranked as category A) |
1 |
|
Results published in a scientific journal (ranked as category B) |
0,9 |
|
Results published in a scientific journal (ranked as category C) |
0.7 |
|
Results published in a scientific journal (ranked as category D) Results published in a scientific journal not listed |
0,6 |
|
Results not published in a scientific journal |
0,5 |
If the results of a study have been published in a scientific journal, the study is weighted higher because this is seen as a further assurance of the quality of the study. The reason being that publication in a journal means that several researchers have evaluated the results of the study prior to publication. However, results of new interventions are more likely to be published than results supporting existing knowledge, and consequently, there is certain publication bias. Furthermore, it takes time before results are published, and consequently, older studies are more likely to have been published than more recent ones. Consequently, there is only a minor difference in weighting with respect to whether results have been published.
You can see how the different scientific journals are ranked here:
Ranking of scientific journals (pdf) (opens in new tab).