Regulatory Guidances

In the 1990s, two noninferiority (NI) guideline documents were released by regulatory agencies: one from U�S� Food and Drug Administration (FDA) (1992) and the other one from the European Agency for the Evaluation of Medicinal Products, Committee for Proprietary Medicinal Products (EMEA/ CPMP) (1997)� These two documents give advice on the NI margin in the anti-infective area with a binary endpoint� Since then, numerous guidelines regarding equivalence trials or NI trials have been released� Treadwell (2011) reviewed and summarized 12 publically available guidelines/documents (mostly from regulatory agencies) for individual trials that describe themselves as either equivalence trials or NI trials:

• Two documents from the International Conference on Harmonization (ICH): ICH E9 (1998) and ICH E10 (2001)

• Four documents from the European Medicines Agency (EMEA): EMEA/CPMP (2000a, 2000b, 2003, 2005)

• One document from each of the following regulatory agencies: a� Japan Pharmaceuticals and Medical Devices Agency (Japan PMDA

2008) b� The Australian Pharmaceutical Benefits Advisory Committee

(Australian PBAC 2008) c� The U�S� Food and Drug Administration (U�S� FDA 2010) d� The India Central Drugs Standard Control Organisation (India

CDSCO 2010) • Two documents from collaborative academic groups: Gomberg-

Maitland, Frison, and Halperin (2003) and Piaggio et al� (2006)

This undertaking is part of research conducted by the Evidence-based Practice Center (EPC) Workgroup (Treadwell, Uhl, Tipton, et al� 2012) with the objective to provide guidance on how to manage the concepts of equivalence and NI in the context of systematic reviews� One of the EMEA/CPMP

documents specifically deals with the choice of NI margin (2005), while the U�S� FDA (2010) document provides guidance specifically for NI trials�

In this chapter, four regulatory documents will be discussed: U�S� FDA (1992), EMEA/CPMP (1997), EMEA/CPMP (2005), and U�S� FDA (2010)� The focus is on determining the NI margin� The first two documents will be discussed in Section 9�2, and the last two documents will be discussed in Sections 9�3 and 9�4, respectively� Additional guideline documents are discussed in Section 9�2�3, including two FDA Anti-Infective Drugs Advisory Committee meetings held in 2002 and 2008�

9.2.1 FDA and CPMP Guidelines

In 1992, the Division of Anti-Infective Drug Products of the FDA issued a points-to-consider document for the development and labeling of antimicrobials (U�S� FDA 1992 pp� 20-21)� It stated the following:

For primary clinical or microbiologic effectiveness endpoints with values greater than 90 percent for the better of the two drugs, a confidence interval that crosses zero and remains within the lower bound delta of –0�10 or less will usually be required to establish equivalence� For primary clinical or microbiologic effectiveness endpoints with values of 80 percent to 89 percent for the better of the two drugs, a confidence interval that crosses zero and remains within the lower bound delta of –0�15 or less will usually be required to establish equivalence� For primary clinical or microbiologic effectiveness endpoints with values of 70 percent to 79 percent for the better of the two drugs, a confidence interval that crosses zero and remains within the lower bound delta of –0�20 or less will usually be required to establish equivalence�

Since the 1990s, the vast majority of comparative trials submitted to the FDA used this so-called “step function” for selecting deltas (Powers et al� 2002)�

In 1997, the CPMP issued a similar guideline (EMEA/CPMP 1997) for the evaluation of antibacterial substances� This guideline proposed an NI margin of 0�1 throughout, but also indicated that a smaller δ may be necessary for very high (>90 percent) response rates� It did not, however, specify how much smaller δ should be (Rohmel 1998, 2001)�

When testing the NI hypothesis given in by Equation 1�3a in Section 1�7 of Chapter 1, both guidelines recommended using the lower limit of the 95% confidence interval (CI) for the treatment difference (T – S) in comparison to –δ� More specifically, the null hypothesis is rejected if the lower limit exceeds –δ� The FDA guideline also required the upper limit of the 95% CI to exceed 0, which was not the case for the CPMP guideline�

Rohmel (2001) contrasted the two guidelines in terms of the margin in Figure 1 of his paper� This figure is reproduced here as Figure 9�1, where q and p correspond to T and S, respectively� Rohmel extended the range of p to 0�5 in Figure 9�1b for the FDA margin and arbitrarily connected the points (p = 0�9, q = 0�8) with (1�0, 1�0) by a straight line in Figure 9�1b for the CPMP margin� The thick lines in Figure 9�1 are the boundary of a subspace in the lower-right corner of the full parameter space� This is called the null space H0� The only point for which both margins (FDA and CPMP) coincide is p = 0�9, q = 0�8, where δ = 0�1� For all other points, the CPMP margin is smaller than the FDA margin�

Note that no explicit NI margin was recommended in a more recent EMEA/CPMP guideline (EMEA/CPMP 2011) or one issued by the U�S� FDA (see Section 9�2�3)�

9.2.2 Varying NI Margin with S

As noted in Section 9�2�1, the CPMP guideline proposed an NI margin of 0�1 throughout, but also indicates that a smaller δ may be necessary for very high (>90 percent) response rates� The reason for a smaller δ when the response rates are high is apparently due to the concern of doubling the failure rate (i�e�, T = 2S)� See, for example, Section 4�2 in Chapter 4 and the following statements excerpted from DiNubile et al� (2012 pp� 1055-1056):

For binary outcomes, when success rates approach 90%, [NI] margins of approximately 10% no longer seem appropriate as they imply that failure rates twice as high as that of the standard are acceptable�

…in circumstances where the standard of care reliably produces response rates of approximately 95%, [an NI] margin as small as 5% accepts a possible doubling of the failure rate with the new therapy�

There are two drawbacks with the FDA margin: (1) the margin as a function of S is not smooth, and (2) the margin is random, as it depends on the observed S (as well as the observed T)� These lead to (1) an erratic behavior of the power curve (Rohmel 1998), and (2) a serious inflation of the Type I error rate (Rohmel 2001)� Rohmel (2001) and Phillips (2003) proposed a smooth function of S as the NI margin� However, these proposals do not take the effect size (i�e�, S – P) into consideration as suggested in Section 2�2 in Chapter 2 and by ICH E10 (2001) and the U�S� FDA (2010)� One alternative to having a smooth varying NI margin is by formulating and testing the NI hypotheses using the odds ratio as the metric, as given by Equation 4�1 in Section 4�3 of Chapter 4�

9.2.3 Additional FDA Guidance Documents and Advisory Committee Meetings

In consideration of the ICH E10 guideline, in February 2001, the U�S� FDA stated on its website that the step-function method is no longer in use and that detailed guidelines on specifying deltas is under development (Powers et al� 2002)� The revised points-to-consider document (U�S� FDA 2001, pp� 20-21) stated the following:

The sliding scale method for determination of delta previously included in this document is no longer in use� Further general information on specifying delta can be found in the 1998 Draft Guidance on Developing Antimicrobial Drugs – General Considerations for Clinical Trials (Section XX�A�6) as well as in the 2000 ICH E10 document (section 1�5�1�1 entitled “Historical Evidence of Sensitivity to Drug Effects and Choosing the Noninferiority Margin”)�

Detailed guidance about delta specification is currently under development� Sponsors are strongly encouraged to specify a delta in their protocols and provide a rationale for that choice� Consultation with the Review division is recommended�

The 1998 Draft Guidance (U�S� FDA 1998) emphasized using a “two-tailed 95% confidence interval around the difference in outcomes” approach to determine equivalence between two products� It stated in Section XX�A�6 (“Issues in Similarity, or Equivalence Trial Design”) that “The lack of a statistically significant difference should not be used as evidence of similarity�” However, there is no guideline on how to determine the delta�

In February 2002, the FDA held an Anti-Infective Drugs Advisory Committee meeting on an approach for selecting delta in NI (equivalence)

clinical trials (U�S� FDA 2002)� Many consensuses were reached in the meeting (Shlaes 2002), such as the following:

• Infectious disease indications are different and it is impractical to have a common statistical requirement across such indications�

• The issue of “biocreep” could be addressed by (1) assuring the use of appropriate comparator agents and (2) in certain cases in which a placebo effect might be strong, the use of a three-arm trial that uses a placebo control with early withdrawal�

It is unlikely that the historical data will support the determination of an NI margin for the indications of (1) acute bacterial sinusitis, (2) acute bacterial exacerbation of chronic bronchitis, and (3) acute bacterial otitis media (U�S� FDA 2010)�

In November 2008, the FDA held another Anti-Infective Drugs Advisory Committee meeting on the justifications of the NI margin for complicated skin and skin structure infections (SSSI) (U�S� FDA 2008)� The committee recommended that NI trials are acceptable in complicated SSSI� However, the committee recommended against using NI trials in uncomplicated SSSI�

The 2005 EMEA/CPMP document entitled “Guideline on the Choice of the Non-Inferiority Margin” is only 11 pages long, including one cover page and one page for the table of contents, leaving only 9 pages for the actual contents� The document describes five situations where an NI trial might be performed as opposed to, or in addition to, a superiority trial over placebo:

1� Applications based upon essential similarity in areas where bioequivalence studies are not possible (e�g�, modified release products or topical preparations)

2� Products with a potential safety advantage over the standard might require an efficacy comparison to the standard to allow a risk-benefit assessment to be made

3� Cases where a direct comparison against the active comparator is needed to help make a risk-benefit assessment

4� Cases where no important loss of efficacy compared to the active comparator would be acceptable

5� Disease areas where the use of a placebo arm is not possible and an active-control trial is used to demonstrate the efficacy of the test product

In situations 2 and 3, the objective of the NI trial is to evaluate the efficacy of T relative to S to assess the risk-benefit ratio� Such an objective is discussed in Section 4 of the document and corresponds to Objective 2 stated

in Section 8�2�1 of Chapter 8� Situation 5 is in disease areas where the use of a placebo arm is not possible (presumably for ethical reasons), and an active-control trial is used to demonstrate the efficacy of the test product by indirect comparison� This corresponds to Objective 1 stated in Section 8�2�1 of Chapter 8� These two objectives are also discussed in Section 2�4 of Chapter 2�

The document emphasizes that the aim of the trial should be precisely defined so that the NI margin may be chosen accordingly� This is in line with the proposed choice of NI margin in Sections 2�2 and 2�4 of Chapter 2, where the choice of ε depends on the study objective� The document states that trials are generally labeled NI trials if they are not aiming to show superiority over the reference� As methodologies showing indirect comparison of T with P emerge, the document points out that (1) “demonstrating [NI]” is not considered to be a sufficiently detailed objective for a trial, and (2) the conclusions of the trial should not be that “[NI]” has been demonstrated, but some more precise statement reflecting the objectives of the trial� Perhaps NI trials with Objective 1 should be labeled “efficacy NI trials�”

Regarding specifying the NI margin as a small percentage of the effect size (see Section 2�2 of Chapter 2), the 2005 EMEA/CPMP document states the following:

• It is not appropriate to define the [NI] margin as a proportion of the difference between the active comparator and placebo� Such ideas were formulated with the aim of ensuring that the test product was superior to (a putative) placebo; however, they may not achieve this purpose� If the reference product has a large advantage over placebo, this does not mean that large differences are unimportant; it just means that the reference product is very efficacious (page 5)�

• Alternatively, the aim may be to provide data to show that there is no important loss of efficacy if the test product is used instead of the reference� This is probably the most common aim of [NI] trials� The choice of delta for such an objective cannot be obtained by only looking at past trials of the comparator against placebo� Ideas such as choosing delta to be a percentage of the expected difference between active and placebo have been advocated, but this is not considered an acceptable justification for the choice� Such ideas were principally formulated to ensure that the reference product was superior to placebo, but this has already been addressed in section III of this document� To adequately choose delta, an informed decision must be taken, supported by evidence of what is considered an unimportant difference in the particular disease area (page 8)�

• The main point is that the aim of the trial should be precisely defined� Following that, a choice for delta should be made, supported by evidence, based upon the precise objectives� This evidence will not

solely come from past trials of the comparator against placebo� Of course, the final choice must always be at least as small as the value derived from the considerations of section III (page 9)�

On one hand, the document appears to recognize the role of the effect size (i�e�, S – P) in determining the NI margin� On the other hand, it is against using a fraction of the effect size as the NI margin, apparently due to the concern that the margin may be too large when the effect size is very large, even though the concept paper on choice of delta (EMEA/CPMP 1999) suggested a delta of one-half or one-third the effect size in some situations (see Item 10 in Section 1�4 of Chapter 1)�

The document provides general considerations when choosing the NI margin� However, no explicit guidance on how to determine the NI margin is given� This is in contrast with the FDA draft guidance to be discussed next, which gives explicit guidance�

The FDA draft guidance entitled “Guidance for Industry: Non-Inferiority Clinical Trials” was released for public comments in March 2010 and has yet to be finalized� It is 63 pages long, excluding the cover page and the table of contents, so it is seven times as long as the EMEA/CPMP guideline� The FDA document provides explicit guidance on how to set the NI margin, including four examples� It recommends first determining the value for M1, defined as “treatment effect of the active comparator” (i�e�, how well the active-control treatment is expected to work), and then calculating the value for M2, referred to as the clinical margin, by taking a fraction of M1�

Using “S” instead of “C” as the active control, the null hypothesis is stated as

H : S T M0 − ≥ (9�1)

versus the alternative hypothesis

H : S T Ma − <

where M is the NI margin that corresponds to δ and is set to M1 or M2� The null hypothesis given by Equation 9�1 is rejected if the upper limit of the 95% CI for (S – T) is less than M, or equivalently, the lower limit of the 95% CI for (T – S) is greater than –M�

Although it is not explicitly stated, it appears that M1 = S – P� Taking a fraction of M1 as M2 is in line with the NI margin given by Equation 2�1 in Chapter 2� On the other hand, M1 looks more like an estimate of S – P� In fact, M1 is typically determined as the lower limit of the 95% confidence for the effect size in the historical trial, with discounting if needed

(see Section 2�5�2 of Chapter 2)� For example, if 10% discounting is used (i�e�, γ = 0�9), then

M 0�9 lower limit of the95% CI for (S P)1 h= × −

where (S – P)h denotes the effect size in the historical studies under the fixedeffect model (FEM) (see Section 7�3 of Chapter 7) or the global mean under the random-effects model (REM) (see Section 7�4 of Chapter 7), although this expression for M1 is not explicitly stated in the draft guidance� A typical value for M2 is often 50% of M1 (i�e�, ε = 0�5), at least partly because the sample sizes needed to rule out a smaller loss become impractically large (U�S� FDA 2010)�

Although M1 and M2 depend on the historical data, they are considered fixed in Equation 9�1� Therefore, the hypothesis testing involving M1 and M2 is called the FEM (see Section 5�3 of Chapter 5)� Without formulating the hypotheses involving the parameters of the historical data as in Section 5�4 of Chapter 5, the description of the synthesis method in the draft guidance is not clear� To contrast the differences between the two methods, the draft guidance refers to Example 1(B)�

The draft guidance recommends the fixed-margin approach for ensuring that the test drug has an effect greater than placebo (i�e�, the NI margin M1 is ruled out), but recommends the synthesis approach for ruling out the clinical margin M2� Strictly speaking, since M2 is fixed, one cannot rule out a fixed margin such as M2 using the synthesis approach�

Australia Pharmaceutical Benefits Advisory Committee (2008)� Guidelines for Preparing Submissions to the Pharmaceutical Benefits Advisory Committee (Version 4�3) Barton ACT: Commonwealth of Australia; Dec� 2008� p� 300� https:// www�pbs�gov�au/industry/listing/elements/pbac-guidelines/PBAC4�3�2�pdf (Accessed: August 25, 2013)�

DiNubile MJ, Sklar P, Lupinacci RJ, and Eron JJ Jr (2012)� Paradoxical Interpretations of Noninferiority Studies: Violating the Excluded Middle� Future Virology, 7(11), 1055-1063�

European Agency for the Evaluation of Medicinal Products, Committee for Proprietary Medicinal Products (1997)� Evaluation of New Anti-Bacterial Products�

European Agency for the Evaluation of Medicinal Products, Committee for Proprietary Medicinal Products (1999)� Concept Paper on Points to Consider: Choice of Delta� CPMP/EWP/2158/99� https://www�f-mri�org/upload / module-5/STAT_CHMP2158_Delta_choice�pdf (Accessed: September 8, 2013)�

European Agency for the Evaluation of Medicinal Products, Committee for Proprietary Medicinal Products (2000a)� Points to Consider on Switching Between Superiority and Noninferiority� https://www�ema�europa�eu/docs /en_GB/document_library/Scientific_guideline/2009/09/WC500003658�pdf (Accessed: August 25, 2013)�

European Agency for the Evaluation of Medicinal Products, Committee for Proprietary Medicinal Products (2000b)� Note for Guidance on the Investigation of Bioavailability and Bioequivalence� London (U�K�); December 14, 2000� https://www�ema�europa�eu/docs/en_GB/document_library/Scientific _guideline/2009/09/WC500003519�pdf�

European Agency for the Evaluation of Medicinal Products, Committee for Proprietary Medicinal Products (2003)� Common Technical Document for the Registration of Pharmaceuticals for Human Use: Clinical Overview and Clinical Summary of Module 2� Module 5: Study Reports� London (UK): July 2003, 44� https://www�ema�europa�eu/docs/en_GB/document_library/Scientific _guideline/2009/09/WC500002723�pdf (Accessed: August 25, 2013)�

European Agency for the Evaluation of Medicinal Products, Committee for Proprietary Medicinal Products (2005)� Guideline on the Choice of the Non-Inferiority Margin EMEA/CPMP/EWP/2158/99� https://www�ema�europa�eu/docs /en_GB/document_library/Scientific_guideline/2009/09/WC500003636�pdf (Accessed: August 25, 2013)�

European Agency for the Evaluation of Medicinal Products, Committee for Proprietary Medicinal Products (2011)� Guideline on the Evaluation of Medicinal Products Indicated for Treatment of Bacterial Infections� https://www�ema�europa�eu /docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003417 �pdf (Accessed: April 30, 2014)�

Gomberg-Maitland M, Frison L, and Halperin JL (2003)� Active-control Clinical Trials to Establish Equivalence or Noninferiority: Methodological and Statistical Concepts Linked to Quality� American Heart Journal; 146(3):398-403�

India Central Drugs Standard Control Organization (2010)� Guidance for Industry on Preparation of Common Technical Document for Import/Manufacture and Marketing Approval of New Drugs for Human Use (New Drug Application – NDA), New Delhi; November 2010, 110� https://cdsco�nic�in/CTD _Guidance%20-Final�pdf (Accessed: August 25, 2013)�

International Conference on Harmonization (ICH) E9 Guideline (1998)� Statistical Principles for Clinical Trials� https://www�fda�gov/downloads/Drugs /GuidanceComplianceRegulatoryInformation/Guidances/UCM073137�pdf (Accessed: September 27, 2012)�

International Conference on Harmonization (ICH) E10 Guideline (2001)� Choice of Control Groups in Clinical Trials� https://www�fda�gov/downloads/Drugs /GuidanceComplianceRegulatoryInformation/Guidances/UCM073139�pdf (Accessed: September 27, 2012)�

Japan Pharmaceuticals and Medical Devices Agency (2008)� Points to Be Considered by the Review Staff Involved in the Evaluation Process of New Drugs� Final� April 17, 2008� https://www�pmda�go�jp/english/service/pdf/review/points �pdf (Accessed: May 1, 2014)�

Phillips KF (2003)� A New Test of Non-inferiority for Anti-infective Trials� Statistics in Medicine; 22:201-212�

Piaggio G, Elbourne DR, Altman DG, Pocock SJ, and Evans SJ (2006)� Reporting of Noninferiority and Equivalence Randomized Trials: An Extension of the CONSORT Statement� Journal of American Medical Association, 295:1152-1160�

Powers JH, Ross DB, Brittain E, Albrecht R, and Goldberger MJ (2002)� The United States Food and Drug Administration and Noninferiority Margins in Clinical Trials of Antimicrobial Agents� Clinical Infectious Diseases; 34:879-881�

Rohmel J (1998)� Therapeutic Equivalence Investigations: Statistical Considerations� Statistics in Medicine; 17:1703-1714�

Rohmel J (2001)� Statistical Considerations of FDA and CPMP Rules for the Investigation of New Anti-bacterial Products� Statistics in Medicine; 20:2561-2571�

Shlaes DM (2002)� Reply to comments by David N� Gilbert and John E� Edwards, Jr� on the paper: Shlaes DM, Moellering RC, Jr� The United States Food and Drug Administration and the end of antibiotics [letter]� Clin Infect Dis 2002; 34: 420-2� Clinical Infectious Diseases; 35:216-217�

Treadwell J, Uhl S, Tipton K, Singh S, Santaguida L, Sun X, Berkman N, Viswanathan M, Coleman C, Shamliyan T, Wang S, Ramakrishnan R, and Elshaug A (2012)� Assessing Equivalence and Noninferiority� Methods Research Report� (Prepared by the EPC Workgroup under Contract No� 290-2007-10063�) AHRQ Publication No� 12-EHC045-EF� Rockville, MD: Agency for Healthcare Research and Quality, June 2012� https://www�effectivehealthcare�ahrq�gov/ehc/products/365/1154 /Assessing-Equivalence-and-Noninferiority_FinalReport_20120613�pdf (Accessed: March 1, 2014)�

Treadwell JR (2011)� Methods Project 1: Existing Guidance for Individual Trials� Plymouth Meeting, PA: ECRI Institute; June 29, 2011, 39�

U�S� Food and Drug Administration (1992), Division of Anti-infective Drug Products� Points-to-Consider: Clinical Development and Labeling of Anti-Infective Drug Products (Rescinded February 2001)� https://www�fda�gov/ohrms/dockets /ac/02/briefing/3837b1_09_1992%20version�pdf (Accessed: September 2, 2013)�

U�S� Food and Drug Administration (1998)� Draft Guidance for Industry: Developing Antimicrobial Drugs – General Considerations of Clinical Trials� July 1998� https://www�fda�gov/downloads/Drugs/GuidanceComplianceRegulatory Information/=Guidances/ucm070983�pdf (Accessed: September 4, 2013)�

U�S� Food and Drug Administration (2001), Division of Anti-infective Drug Products� Points-to-Consider: Clinical Development and Labeling of Anti-Infective Drug Products� Disclaimer of 1992 Points-to-Consider Document� https://www�fda�gov /ohrms/dockets/ac/02/briefing/3837b1_11_points%20to%20consider%20 GFI�pdf (Accessed: September 2, 2013)�

U�S� Food and Drug Administration (2002), Division of Anti-Infective Drug Products Advisory Committee Meeting, February 19-20, 2002� https://www�fda�gov /ohrms/dockets/ac/cder02�htm#Anti-Infective (Accessed: September 2, 2013)�

U�S� Food and Drug Administration (2010)� Guidance for Industry – Antibacterial Drug Products: Use of Non-inferiority Studies to Support Approval� https://www �fda�gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/ Guidances/ucm070951�pdf (Accessed: August 16, 2014)�

U�S� Food and Drug Administration (2008), Division of Anti-Infective Drug Products Advisory Committee Meeting, November 19-20, 2008� (https://www�fda�gov /ohrms/dockets/ac/cder08�html#AntiInfective (Accessed: September 2, 2013)�

U�S� Food and Drug Administration (2010)� Draft Guidance for Industry: Non-inferiority Clinical Trials� https://www�fda�gov/downloads/Drugs /GuidanceComplianceRegulatoryInformation/Guidances/UCM202140�pdf (Accessed: August 25, 2013)�