Drug ratings for new medicines

For the last ten years, the Geneesmiddelenbulletin has been publishing evaluations of new medicines in the form of drug ratings (Pilwaarderingen in Dutch). The editorial committee has decided to change the rating system by assigning only three grades instead of the five it used in the past. Each evaluation is visualised in a cartoon (dr Frits). It also decided to make more frequent use of the opportunity to reassess previous evaluations in the Bulletin Board section under the heading of Opnieuw bezien (Reassessment). This article discusses the criteria used in the evaluations and presents an example of an assessment of a new drug with its resulting rating (Gebu 2010; 44: 61-64).


Following the example of some of our sister publications of the International Society of Drug Bulletins (ISDB), the Geneesmiddelenbulletin introduced the drug ratings for new medicines ten years ago (Gebu 2000; 34: 111). The rating was indicated by symbols going from ++ to –, providing the reader with an easily recognisable evaluation of the new drug. The box below explains the meaning of the various symbols.

Meaning of symbols used for drug ratings in the Geneesmiddelenbulletin.

++ : important addition to the pharmacotherapeutic repertory

+     : useful drug

+/-  : product of doubtful value or whose value can as yet not be adequately evaluated

–      : product offering no added value

– –   : product posing additional risks and adding no new treatment options

Since then, the Geneesmiddelenbulletin has evaluated over 80 new drugs and assigned ratings to them. The great majority were rated as +/- or -. None were given a ++ rating, and only one was given a — rating. This led the editorial committee to review and revise the evaluation categories. It concluded that the ++ and – – ratings were in practice assigned so rarely that there was no point in retaining them.

This article explains the background of the drug ratings by discussing the evaluation procedure and illustrating this with an example.

Evaluation procedure

General. The editorial committee evaluates new or existing drugs by assessing the balance of their efficacy and side-effects or safety in comparison to the existing standard therapy. The aspects addressed in this evaluation have been extensively discussed in the Geneesmiddelenbulletin publications, so we briefly review only the main ones here. One aspect is the quality of the research that has been done on the drug, as well as the number of published studies. In addition, different factors may be relevant to the ratings for different medicines. This is always explicitly indicated in the articles, and particularly in the assessment of the place of the drug in the pharmaceutical repertory (Plaatsbepaling). In addition to efficacy and side-effects, the evaluation also discusses interactions, contra-indications and use during pregnancy and lactation, as well as price.

Gold standard. The Nieuwe geneesmiddelen (New Drugs) part of the Bulletin Board section only discusses randomised and preferably double-blind trials published in peer-reviewed journals. The experimental design of such randomised clinical trials is regarded as the gold standard for assessing the effect of an intervention, as the findings of this type of research are least likely to be biased by methodological shortcomings. Nevertheless, there may well be certain shortcomings in a study which hamper its interpretation.

Assessing efficacy and side-effects. The Geneesmiddelenbulletin tries to arrive at an independent assessment of the place of a new drug in the therapeutic repertory. The journal subscribes to the Consolidation of the standards of reporting trials (CONSORT) guidelines for reporting on randomised clinical trials and hence on the efficacy of drugs (Gebu 2009; 43: 111-113) and also checks published articles against these guidelines.1-3

As regards reporting about observational studies, particularly research into side-effects of drugs (cohort studies, case-control studies and cross-sectional studies), the Geneesmiddelenbulletin subscribes to the Strengthening the Reporting of OBservational Studies in Epidemiology (STROBE) guidelines.4-6

In principle, the Geneesmiddelenbulletin does not publish results derived purely from abstracts, posters, papers read at conferences, data on file and expert reports, as this material has usually not been verified by independent reviewers. It has happened more than once that an abstract subsequently submitted as an article presented slightly different data, or that it was not published as an article at all. Moreover, the information provided is in many cases insufficient for an adequate interpretation of the results.

Interpretation of published articles. It turns out that various conditions that are described in the CONSORT guidelines and that peer-reviewed journals subscribe to, are not always consistently adhered to by their editorial committees when assessing articles. The editorial committee of the Geneesmiddelenbulletin frequently finds that data that are essential for the interpretation of the findings are missing from research reports. Relevant aspects include a description of the randomisation procedure, information on drop-out, sufficient descriptions of side-effects and statistical analysis of the side-effects (Gebu 2009; 43: 111-113).

The process of interpreting drug research findings focuses on methodological factors affecting the internal and external validity.7-8 The internal validity of a study refers to the extent to which the results describe the actual reality (in this case in the prescriber’s practice). The extent to which the results deviate from and thus distort reality is called bias. The external validity of a study refers to the question whether and to what extent the findings can be applied to patients outside the study sample, in other words whether the study results can be generalised. The main forms of bias are selection bias, information bias and confounding bias (see e.g.  Gebu 1999; 33: 127-134). Although a detailed discussion of these concepts is beyond the scope of this article, we would like to briefly mention some of the main aspects.

Generalisability is an important aspect of a study. One question may be whether the results of a study in a tertiary care setting are also valid for primary care patients. The inclusion and exclusion criteria used to establish the study sample can indicate whether the study sample is comparable to one’s own patients. If the study made use of questionnaires, it is important to know whether these have been validated for the relevant population. Results should not only be statistically significant, but also clinically relevant. Statistical significance is especially determined by the size of the study sample, while clinical relevance relates to the effect size that can be achieved with the drug being studied. The fact that a drug has a novel mode of action may be interesting, but offers no guarantee that it will produce better results than the standard therapy. Studies should also include a sufficient number of patients, who have been followed for a sufficient period of time. The research methodology has to be appropriate if the study is to be given rating better than +/-. Studies with a non-inferiority design, which allow one to conclude whether the new drug is inferior to the drug it is compared with, are the least desirable type from a methodological point of view. In studies concerning preventive treatment it is important that hard (i.e. clinically relevant and accurately measurable) endpoints are studied and reported. It is also important whether findings of studies into long-term side-effects have been published. And of course it is important whether the efficacy of the new drug has been compared with that of the standard therapy.

These and other criteria are used by the editorial committee of the Geneesmiddelenbulletin to assign a rating to a new drug. The editorial committee has decided to reduce the rating categories to three (see the box below).


These new ratings are presented in the form of the above dr Frits cartoons (created by artist Mathijs Wansink), which have already been used a few times in issues of the Geneesmiddelenbulletin.

The editorial committee has also decided to make more frequent use of the opportunity to reassess previous evaluations in the Bulletin Board section under the heading of Opnieuw bezien (Reassessment). The 2013 issues are reviewing the ratings assigned in 2000, regarding apomorphine, brinzolamide, montelukast, risedronic acid and imiquimod. These drugs were given a +/- rating at the time, except for brinzolamide, which was not given any rating. The reassessed drugs are once again given a rating.

An example of a drug rating: saxagliptine

Gebu 2010; 44: 49-55 discussed saxagliptine, but without assigning a rating to this drug. The presentation of the main data below allows us to show how a drug rating is established.

Saxagliptine belongs to the category of dipeptidylpeptidase (DPP)-4 inhibitors and is the third drug of this group to have been authorised for the treatment of type 2 diabetes mellitus, as an adjunct to metformin, a sulfonylureum derivative or a thiazolidinedion derivate, if monotherapy with these drugs achieve insufficient glycaemic control.9

Clinical research. Three randomised, double-blind, placebo-controlled trials published in peer-reviewed journals have compared the efficacy of saxagliptine with that of metformin or glibenclamide over a period of 24 weeks. 10-12 A proportion of the randomised patients used saxagliptine off-label, as they had not previously been treated with an oral blood glucose lowering agent. The findings showed that the addition of saxagliptine to metformin reduced the HbA1c level by an additional 0.5 to 0.83% compared to the group who had had a placebo added to their metformin treatment.

Side-effects. Diarrhoea occurred in 5.5 to 9.6% of the patients who used saxagliptine. The researchers did not check whether side-effects occurred more frequently than with the placebo. Other reported side-effects included nasopharyngitis, upper airway infections and urinary tract infections, while headaches also occur.10-12

Interactions. Saxagliptine is a substrate for CYP3A4, so interactions can be expected with powerful inhibitors of this enzyme, such as diltiazem and ketoconazole, or inductors like rifampicin.9

Contra-indications. In view of the limited experience with DPP-4 inhibitors in patients with moderate or severe kidney or liver failure or in patients younger than 18 or older than 75 years, the use of these agents by these patients is not recommended.9

Use during pregnancy or lactation. In view of the lack of data, DPP-4 inhibitors should not be used during pregnancy or lactation.

Methodological limitations of the studies into saxagliptine.

Many of the studies used a single-blind placebo ’lead-in’ phase lasting one or a few weeks, in which patients’ compliance with a diet and exercise programme was assessed. Only those patients with a high level of compliance (>80%) were allowed to participate in the rest of the study. Strict exclusion criteria were applied, and patients were excluded if they had a history or heart, liver, kidney or psychiatric disorders. In many of the studies, a considerable proportion of the randomised patients were excluded in this period, while some of the remaining patients dropped out due to side-effects, poor compliance or lack of effect. In some of the studies, up to about 40% of the patients were thereby lost to follow-up.

The analytical method used in the studies, the so-called Last Observation Carried Forward (LOCF) method, is not the most obvious method to use if a study has a high drop-out rate.

The clinical relevance of the differences in HbA1c levels eventually found between the different drugs is questionable.

In conclusion, the studies used highly selected study populations, and subsequently used enrichment phases (including only patients with high compliance), with substantial drop-out rates and with questionable outcomes in terms of the clinical relevance of the differences found. This means that the external validity, that is, the generalisability of the results, as well as the internal validity, are limited.

Place in the pharmacotherapeutic repertory. Saxagliptine has only been compared with placebo as an adjunct to metformin or glibenclamide in short-term studies lasting no more than 24 weeks. Some of the patients in these studies used the drug off-label, as they had not previously been treated with antidiabetic agents as a monotherapy. The registered indication for the drug ignores the recommendation that failure of metformin monotherapy should be followed by the addition of a sulfonylureum derivative, based on efficacy and side-effects (Gebu 2007; 41: 111-112).13 There have been no studies comparing the addition of saxagliptine to such a combination of a sulfonylureum derivative and metformin with the addition of placebo, nor studies to compare the combination with one of the drugs as a monotherapy in combination with saxagliptine. This means that important information required for scientifically sound decisions about prescribing saxagliptine in routine practice is lacking. This type of gap is usually not mentioned in marketing campaigns.


The studies used highly selected samples of patients who had already demonstrated good compliance. This means that the study sample approached the kind of group normally selected for per-protocol analysis. This type of analysis usually yields results that are more favourable to the investigated drug than intention-to-treat analysis, which is closer to the practical situation where patients frequently comply less strictly with their medication regimes. This compromises the external validity.

As regards efficacy, it is questionable whether the additional 0.5 to 0.83% lowering of the surrogate parameter HbA1c relative to placebo is clinically relevant (Gebu 2010; 44: 51).

DPP-4 inhibitors influence several physiological mechanisms in the human body, and can therefore potentially have a wide range of side-effects. In such a case, if only data are available about short-term side-effects, there is all the more reason to remain alert to any longer-term effects.


The rating system for new medicines in the New drugs part of the Bulletin Board section has been reduced from five to three categories: + for a valuable drug that is expected to represent a useful addition to the existing pharmacotherapeutic options; +/- for a product of doubtful value or a drug whose value can as yet not be adequately evaluated and – for a product offering no added value, usually a me-too product. Finally, a red card is assigned to a product that poses additional risks and adds nothing to the existing therapeutic options. The reason for this change was that the ++ and – – ratings were in practice assigned so rarely that there was no point in retaining them. The ratings are now presented through the dr Frits cartoons.

The evaluation is based on several criteria, the most important being the quality of the research and the number of studies that have been performed and published. A particularly important issue is whether the internal and external validity are sufficiently proven so that the results can be applied in the prescribers’ routine practice. The efficacy of the new drug must be sufficiently established and the drug must have been compared with the current standard therapy. Another important issue is that of side-effects, and especially the question whether rare and long-term side-effects have been satisfactorily studied.

So far, the Geneesmiddelenbulletin has evaluated and assigned ratings to over 80 new drugs. In the near future, we intend to review a number of the evaluations of drugs that were newly introduced in the year 2000 under the heading of Opnieuw bezien (Reassessment).

Key words: drug ratings, new medicines, saxagliptine, cartoons, dr Frits

References: these are available at www.geneesmiddelenbulletin.nl

Generic name Brand name®
apomorphine APO-go
brinzolamide Azopt
diltiazem generic, Tildiem
glibenclamide generic
imiquimod Aldara
ketoconazole generic, Nizoral
metformin generic
montelukast Singulair
rifampicin generic, Rifadin
risedronic acid Actonel
saxagliptine Onglyza


  1. Verslaglegging van gerandomiseerd medisch-wetenschappelijk onderzoek volgens een standaardmethode; de ‘Consolidation of the standards of reporting trials’ (CONSORT) [redactionele kanttekeningen]. Ned Tijdschr Geneeskd 1998; 142: 1089-1091.
  2. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996; 276: 637-639.
  3. The CONSORT group. The CONSORT statement. Via: http://www.consort-statement.org/consort-statement/overview0/.
  4. Elm E von, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, et al. The strenghtening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med 2007; 147: 573-577. Of via URL: http://www.strobe-statement.org/index.php?id=strobe-publications.
  5. Vandenbroucke JP, Elm E von, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al. Strenghtening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration. PLoS Med 2007; 4: 1628-1654. Of via URL: http://www.strobe-statement.org/index.php?id=strobe-publications.
  6. Stehouwer CDA. Rapportage van observationeel onderzoek: nuttige en welkome aanbevelingen ter verbetering [commentaar]. Ned Tijdschr Geneeskd 2008; 152: 182-184.
  7. Hofman A, Grobbee DE, Lubsen J (red.). Klinische epidemiologie. Utrecht: Bunge, 1996.
  8. Bouter LM, Dongen MCJM van, Zielhuis GA. Epidemiologisch onderzoek: opzet en interpretatie. Houten: Bohn Stafleu van Loghum, 2010.
  9. Productinformatie saxagliptine (Onglyza®). Via: www.ema.europa.eu, human medicines, EPAR’s.
  10. DeFronzo RA, Hissa MN, Garber AJ, Gross JL, Duan RY, Ravichandran S, et al. The efficacy and safety of saxagliptin when added to metformin therapy in patients with inadequately controlled type 2 diabetes with metformin alone. Diabetes Care 2009; 32: 1649-1655.
  11. Jadzinsky M, Pfützner A, Paz-Pacheco E, Xu Z, Allen E, Chen R; CV181-039 Investigators. Saxagliptin given in combination with metformin as initial therapy improves glycaemic control in patients with type 2 diabetes compared with either monotherapy: a randomized controlled trial. Diabetes Obes Metab 2009; 11: 611-622.
  12. Chacra AR, Tan GH, Apanovitch A, Ravichandran S, List J, Chen R; CV181-040 Investigators. Saxagliptin added to a submaximal dose of sulphonylurea improves glycaemic control compared with uptitration of sulphonylurea in patients with type 2 diabetes: a randomised controlled trial. Int J Clin Pract 2009; 63: 1395-1406.
  13. Rutten GEHM, De Grauw WJC, Nijpels G, Goudswaard AN, Uitewaal PJM, Van der Does FEE, et al. NHG-Standaard ’Diabetes Mellitus’ 2006. Huisarts Wet 2006; 49: 137-152.