How to Reduce Data Variability in Respiratory Clinical Trials
Phil Lake and Dr. Kai Michael-Beeh examine the steps sponsors and study teams can take to improve data quality and reduce data variability in respiratory trials. They also discuss the innovations and trends they expect to see in the industry in the future.
Access Trial Better Podcast from your favorite channels:
Phil Lake and Dr. Kai Michael-Beeh examine the steps sponsors and study teams can take to improve data quality and reduce data variability in respiratory trials. They also discuss the innovations and trends they expect to see in the industry in the future.
How can sponsors and the pharma industry overcome unacceptable data variability from sites? [03:30]
Respiratory measurements can be complicated and challenging if study teams don’t stick to the fundamentals. This means that variability and bad data quality are common issues. Two major factors can contribute to variability: disease-associated factors and effort-dependent factors. Both of these issues can be managed, with strict standardization in the protocol and rigorous training, respectively.
Is a focus on ATS/ERS standards enough to generate research-grade data?[06:34]
ATS/ERS standards are a suitable starting point and principle for quality assurance. However, simply applying these rules is not sufficient. In addition to following these standards, sponsors should implement additional visual inspections and plausibility checks.
How can we use available technology and services to reduce data variability? [09:50]
Available technology and data should be used to guarantee the production of the highest quality data possible. Ensure that investigators are aware of what defines a good quality test in the context of a clinical trial, including how important it is to show changes in lung function. Increased overall communication with investigators decreases variability, as well as the number of patients needed (and ultimately costs.)
How can sponsors and sites best stay on track with projected timelines?[14:35]
Communicating with sites so they can more effectively allocate their resources, especially when a study has fallen behind, can be a crucial improvement. Checks should take place to ensure that third-party providers are ready to begin once the site is initiated.
What’s the real cost or impact to pharma when we produce variable data or run into study delays? [19:18]
Variable data and study delays have a tremendous impact by putting approvals at risk and jeopardizing the reputation of the drug or manufacturer. When studies produce better data, there’s an increased chance of achieving study goals with less patients and lower costs. Better data also improves confidence in negative results.
Any future innovations that will improve data quality? How can we do better as an industry to understanding drug effect? [21:38]
In the future, the ability to measure various aspects of airway function will improve due to the shift towards personalized medicine, with new methods regulated and approved by the FDA.
[Intro]: Welcome back to the Trial Better podcast. This week we’re talking spirometry and the complexities of respiratory drug development. Join host, Phil Lake and guest, Dr. Kai Michael Beeh. as they discuss how to capture quality respiratory data and the obstacles that lead to data variability. So, let’s not waste time and dive into another episode of the Trial Better podcast.
Phil Lake: Hi everyone. This is Phil Lake from ERT and today on one of our Trial Better podcasts, we’re going to talk to Dr. Kai Beeh. from the Insaf Research Institute. I believe, Kai, we’ve been probably working together for, maybe the last 15 or 20 years indirectly. It’s a great pleasure to meet you. I just wanted to start off. If you can actually tell us a little bit about your work as an investigator in the inspectory at a couple of trial sites director and stuff
Dr. Kai-Michael Beeh: Yes. I feel great to work with you on this occasion and great that we are having this podcast and this nice conversation together. So you already mentioned, my profession currently, is I’m a medical director at the Insaf Respiratory Research Institute. By training, I’m a pulmonologist. So I have now more than 20 years experience in clinical medicine and pulmonary medicine. Coincidentally, I also have a little bit more than 20 years now of experience in clinical research. I’ve worked in all parts of academic clinical medicine but also in ambulatory outpatient clinics I have a bit of experience with particular airway diseases.
So for the last 15 years, I have been focusing my work on the performancy conduct of clinical trials. So we’re a clinical research institute where we do all types of trials from, let’s say, phase one to phase four inpatient with respiratory diseases which is mainly asthma and COPD. That is my everyday work and my involvement in the industry, it’s been more or less everything from, let’s say copying paper and providing a copy to basically writing protocals or doing ethics submission, identifying clinical unmet needs in patients, discussing draft targets to finally and ultimately doing or conducting at the patient level, clinical trials. That is more or less, my everyday clinical work.
Phil Lake: Yes, inspectory, you’ve clearly got a broad range of experience. Can you tell me, have you consulted directly with pharma companies and if so, what kind of projects have you been working on?
Dr. Kai-Michael Beeh: We have a long-standing history of operation with different types of pharma companies. We are a clinical site, directly performing clinical site, but we are also involved heavily in actually consulting in respiratory, which goes from everything to actually designing a trial, choosing the right outcomes for them.
I’ve also been involved in external consulting of quality of pulmonary function techs or other parameters. As I said, most of it is in asthma and COPD. Well, I’m quite sad to say that if you look at the current market in asthma and COPD, there’s basically none of the drugs that have been released in the last, let’s say 10 to 15 years where my institution has not been made a part in performing the trials and the level of phase two, phase three or whatever trial.
Phil Lake: Excellent. Now, thank you for that. What are the core challenges that you face as an industry particularly in respiratory drug development? How do we actually deal with unacceptable data variability from sites and basically, what’s your approach in terms of how you can overcome that?
Dr. Kai-Michael Beeh: Well, I have a very, very long experience with doing the particular measurements for spirometry and stuff like that. If I counted it, I think sometimes calculate, probably in my life together with the time when I worked as a student in an outpatient clinic, I think I probably have a record of let’s say 100,000 maybe 120, 000 spirometry which is quite a lot, I assume.
I find the core of these measurements to be pretty complicated and challenging. You don’t stick to the fundamentals of it. Talking about variability, I think variability due to bad data closely is a big issue for the industry. I would identify basically two major factors that lead to data variability. One, of course, is disease-associated. When we talk about airway disease, all of these diseases have a certain fluctuation.
A particular patient with asthma, they have a diurnal or circadian variation and then there’s this thin hair in variability of airflow in somebody with more or less severe asthma or COPD. You can also observe hyper-responsive stations during a maneuver that, for example, by the maneuver itself. They have a triggering of airflow obstruction.
Then we have other factors that contribute to variability, for example, when you describe the impact of a deep inspectory effort, which leads to some degree of bronchodilation. These are the factors which are associated with the disease and they are very difficult to control. It’s not impossible to control them but they are difficult to control.
The second is, of course, performance-associated. The issue with lung function is, most of them, is that they are effort dependent, highly effort dependent which means you need proper instruction on one hand, and you need absolute cooperation and training by the patient on the other hand. That is something which can be very nicely addressed by clear instructions and repeated and clear and sufficient training, not only of staff but also patients, everybody who is involved in performing the test should be rigorously trained.
Addressing issues with disease-associated factors, as I said, may be more complicated, but even that can be addressed for example, by strict standardization within protocol. For example, the time of the day should be consistent; the wash out period to fill the graph must be consistent. And all of these are factors that can be worked on if you design the protocols in a very nice way.
Phil Lake: That’s good. Thank you. I’ve seen at my time in the industry how things are advanced from the initial trials I worked on where we had no kind of centralized data, at least, to allow sites to generate their own data. When you work to eight years, you are a standard. Now, do you think that just focusing on that, is that good enough in terms of generating research-grade data?
Dr. Kai-Michael Beeh: Well, I think it’s a starting point. Eight years, yes, standards are very free to do—It’s like, call it an overarching principle for heading flip-tight or standards or quality assurance. However, I totally agree with you that the simple application of these rules like start of test and end of test of following a typical algorithm is not totally sufficient.
That’s why I think there should also always be something like an additional visual inspection of the trials and particularly, plausibility check. This should be done by somebody who is experienced with the procedure. I’ll give you a very simple example. When ATS criteria would clearly fail to identify a test that has a problem. If you, for example, at a site, you mix up patients, which sometimes happens just by mistake, when you don’t call Mrs. X, you just call Mr. Y.
If you only have an identifier, so no name, it’s all numerized or enumerized. If you don’t have a name, sometimes that happens, you measure the wrong patient with the wrong idea. The ATHA quality criteria would simply say, “Well, that’s a perfect test. However, if you do a plausibility check, you would see that this test has got nothing to do with the other test of this particular patient.
It takes a human being to actually check on some type of algorithm to also check it’s data for plausibility. This is a very simple example where there is a shortcoming with just applying ATS and EOS standards. I think there must be something beyond that and that ideally, is somebody like an external operator that communicates with the sites and particularly communicates when there’s disagreement about the quality of a certain test.
Phil Lake: It’s good to hear you say that because I see very similar things. So again, this is one of the problems we face as an industry when we are dealing with sites who are working on a lot of different projects. I think if they just look at the ATS ERS standards and then send the data off to ERT, they may fail to intercept some of these problems when they’ve actually got an opportunity to make a difference.
Again, I’ve seen instances where a patient has taken a rescue medication use during a visit and obviously fundamentally changed their lung function. Of course, they still meet ATS ERS standards and they send the data off to ERT and really, there’s nothing we can do at that stage.
The data is included in the ITT analysis and that’s it. Like you say, if they do a plausibility check and actually understand that something has changed, they can explore that and then they have the potential to reschedule the visit. Therefore, they avoid the errors in the database, so it’s good to hear you say that. One of the other questions I had is, in terms of the technology and services, how can we use this to help to reduce the variability?
Dr. Kai-Michael Beeh: Well, I think it’s important to use the available technology and particularly, our services when you really focus on respiratory or pulmonary functions outcomes and you want to guarantee the best estimate quality of data. I think the first step is really to get investigators from sites to be aware of what defines a good quality test in the context of a clinical trial.
Because there is, I think, in many investigators who also work as clinicians, there is a misconception. Because they work in their everyday clinical practice, they usually work with these types of tests using that as a diagnostic tool to say more or less, somebody is normal or somebody is not normal. However, they probably will not pay attention to, let’s say, the first definite digit of an ACV1 or AC3.
On the same hand, the same people accept that, for example, a change in 100ml of lung function in the patient with COPD is clinically meaningful. So it just takes bit of, actually, training and instruction to make these people aware of how difficult it can be to show a statistically significant change of lung function in the range of 100ml if you don’t take care that your lung function is as repeatable, as acceptable as possible and you reduce variability.
The best way to do that is, yes, through technology on one hand, and on the other hand, to us external people. Like you do monitor clinical data and the clinical trial, it is important to monitor the quality of lung function which is sometimes a sensitive procedure because doctors are not actually used to letting somebody grade their spirometry, particularly pulmonologists who probably will say, “Hey, guy, you are trying to teach spirometry to me. I’ve been doing this for 20 years,” just as I said in my introduction. A little bit of sensitive conversations at times.
However, I’m frankly convinced as with myself that if you just tell people how important it is to minimize variability to show more change in the lung function which may be clinically variable, I’m pretty sure that everybody will perfectly understand and will undergo this procedure. In short, yes, this technology and these services must be used in every clinical trial that do of pulmonary function that’s an outcome.
This is very, very important and historically, we’ve seen a major shift in the quality of data, of spirometry data, particularly in the past years. And a lot of that is due to the centralization of the process and the external operator. That’s absolutely without doubt.
Phil Lake: I completely agree. When I think back to some of the first studies that I was involved with, looking at some of the first combination products in the market, a lot of that data just wasn’t centralized but the treatment effect was so large that you didn’t really need to variabilities so much. But, we’ve gone through, as an industry, we are now looking at much smaller treatment differences because we have good established medications and standard of care and then, of course, we’ve got all the personalized medication as we move forward to areas like cystic fibrosis. So, trying to understand exactly how each patient response to therapies becomes a lot more important.
Dr. Kai-Michael Beeh: It is important to communicate to investigators also. One thing is, of course, your point is well-taken. Decreasing variability decreases the number of patients. Actually, you need to show, of course, the setting that there’s a small effect. That’s clear and that reduces cost. Well, you could argue, as an investigator, I’m not so much concerned about the cost of fully done. However, the second part which is if you minimize the amount of patients that go into a clinical trial from an ethical point of view, as you are testing investigational drugs which often have an unknown side effect profile if you minimize the number of patients that need to be exposed to a drug where we don’t actually know the full side effects profile. I think it is also an important ethical consideration for doctors that they are obliged to minimize the variability of the data just to minimize the number of subjects which are necessary to answer a specific research question. That’s absolutely true both on the cost side but also on the performance side which affect the investigator or the doctor.
Phil Lake: Yes. That makes a lot of sense. Okay, so I just wanted to ask you about, as we get through the process, the whole process about how we finalize the database and move through to generate or to resolve all those outstanding queries. What obstacles can you see to achieving this last subject, last visit on time and how we best stay on track with productive study timelines.
Dr. Kai-Michael Beeh: Well, I do have a specific view on that– I’m clearly biased as an investigator because I see the problems that we sometimes have to fit the project into our timelines when on the other hand, those who organize the projects do not stick to the timelines. But I think generally, actually, most studies that I’ve been part of in the past were in time and most of them were even closing database before the deadlines.
But nevertheless, there’s always room for improvement. Quite often, time is lost in regulatory procedure, ethics approval, multinational studies or something like that. It is very important that if these problems occur, that there is communication with the sites because sometimes, we can reallocate resources, patients and something like that, we know that this study is going to have a delay of such and such weeks, instead of just saying, “We are about to start but we don’t exactly know the dates.” So, communication is key.
Then, there is also the vendor part. There is the vendor part sometimes where you have a study ready to go and a third-party provider doesn’t have the necessary equipment while everything is ready to start. My view as an investigator is that if you are using tools of third-party vendors and instruments, that the approach of a permanent bidder like companies like Google or Apple now apply. A permanent bidder is not an option for a clinical trial. It is very important that the machinery and the software is ready to go once the site is initiated. That’s also very important. For my responsibility at the site, of course, sometimes it is difficult to identify the correct patient. There are difficult protocols and it’s sometimes also difficult to retain subjects in the study, particularly, the long-term studies and you are having incidents like exacerbations or something like that. I think as the principal of a site, I would like to know at every step of trial planning and conduct, where we stand and what the exact anticipated project timelines are and importantly, if there are any changes that are to be anticipated.
With this information, I can at least do my best to make sure that me and my team can dedicate 100% through the performance at the database log of this particular trial. However, if the communication is not good, sometimes it is always a problem that information get’s lost. That sometimes makes it very difficult.
As we are doing, usually, we are doing multiple trials at the same time to really dedicate the resources to that part of the project that needs it most at this particular stage. But I think communication is actually important.
Phil Lake: No. You raised a good point there. One of the things within the pharma industry we tend to forget is that sites have a lot of competing priorities so again, that lack of communication is often self-defeating. So, if you have delays or other issues and you are unaware, you can’t then juggle your priorities to help meet the commitment which you signed up to. Do you see that commonly?
Dr. Kai-Michael Beeh: Yes. That’s a common problem. We do give our commitments to studies when they ask us to do so but we have a certain timeline then and we make sure that our studies do not directly compete with each other so that we can give 100% to the core studies. However, if one study is six weeks late and the other one is six weeks earlier, the problem is that it’s something we cannot anticipate and I cannot, at that stage, I cannot draw any of my commitments to one of these studies. Then sometimes I just have to say, “Okay, I have to do both at the same time and then of course, I do not have to give 100%, I need to give it 200%. But this is something I cannot control. As somebody who is also—I have to run a business. So, for me, it’s not impossible to say, “We are doing only this one trial and maybe there will be another one.” I need to plan ahead, at least, six months or eight or twelve months with my institution. So that is very important and this is again, where communication comes in.
Phil Lake: Yes. That’s very obvious. One thing I’d like to find out your perspective on as well is your understanding about real cost and the impact to pharma when we see more variable data or delays as we’ve been discussing? What’s your perspective on that?
Dr. Kai-Michael Beeh: It’s tremendous. In the end, you jeopardize the ultimate outcome of a trial or a respiratory trial and you may eventually put a potential approval at risk or at least, impact on the reputation of a drug or it’s manufacturer. But then in the end, you may require additional or repeat trials at substantial costs, millions, sometimes hundreds of millions of dollars. In many medications I work on, as I said before, very small changes can mean a substantial improvement in patients and in patients’ lives.
So what does that mean? Well, every milliliter counts but data variability because of poor tests makes it very difficult to detect the two changes with a drug, with the normal investigation of drugs. The purpose of better data quality is absolutely obvious in my mind. You have a greater likelihood of achieving a study goal with fewer patients at lower costs.
In the case of a negative study, even that gives you confidence as a sponsor that you can readily refute a hypothesis that you have generated because that is not true but the data quality clearly tells you, well, there’s no alternative hypothesis or no other type of error.
Phil Lake: Yes. That’s a good point. I think we all focus on positive trials but if you have a negative result, it’s high variability. You’re never quite certain whether that is a real negative result, the drug doesn’t work or whether it’s just an alignment of chaff.
Dr. Kai-Michael Beeh: In the field where I work, so most times I work at phase two, so sometimes I used to joke to my students or my family and say my business is failure. Most of the studies that I got are negative. I totally agree with you. The best thing about a negative study is that well, it’s so absolutely clearly negative so that everybody, the sponsor or whatever, even an academic researcher will have confidence there’s absolutely no sense in going further. The worst is you have a negative study and you have bad data and you don’t know whether this is any type of statistical error that you have not actually consoled for because of data variability.
Phil Lake: Now, I know exactly. Okay, so are there any innovations you see coming through in the future that you think will improve data quality? So how can we really do better in the industry to better understand the triangular effect?
Dr. Kai-Michael Beeh: Well, I do see a bit of revival or an essence of physiology in early disease which has long been neglected because everybody is just doing genes or TCR or immunology and pieces and stuff like that. We’ve seen a revival of physiology so I think it is important of a study level that we need to extend our instrumentarium to measure various aspects of airway functioning and doing this at a quality level that makes these parameters acceptable for authorities like the FDA and then ultimately, granting approval or a legal claim for drug.
I think that in the age of everybody talking about putting medicine, a simple SED1 only approach cannot remain our standards forever. As a former academic and a researcher, I think we should strive to identify sample eventual measures of airway function and make them finally suitable for registration and trials in large numbers of patients.
And then, of course, in this respect, we depend on co-operations with industry and vendors like you. My advice is that even if you are a service-driven company, don’t forget the science. I think that is very important and in the end, I firmly believe that such co-operations between industry vendors and academics or researchers will be simply a win-win for all stakeholders. That is my absolute conviction.
Phil Lake: I agree. I think with that kind of approach, we are likely to get a better understanding and potentially, better drugs as we go forward. Okay. So, really that’s the questions I really wanted to ask you. I think it’s been a great level of insight in terms of your clear experience in the industry. Are there any final thoughts you wanted to leave us with, Kai?
Dr. Kai-Michael Beeh: Well, I think at this time my state of view. Something that I always like to mention or emphasize as a clinical researcher who is running a site and who is still indirect contact with a lot of research subjects and patients, I would like to emphasize still the importance of the site. I will do this over and over again. And it’s important to notice everybody who’s outside maybe not doing the stuff as the site is that the landscape and environment for clinical trials in the last 20 years has changed dramatically and the burden on sites has increased and on average, the conduct of trials has become even more challenging.
So, yes, I do welcome many of the changes, particularly when the quality of research is improved, there’s absolutely no doubt about that. So, such as the use of external ovaries for the key clinical procedures. But, of course, I still want to be regarded as a partner in this process and I like that investigative sites raise their voice and also be heard in the process of trial planning.
And I’m convinced that sponsors and vendors can benefit a lot from the experience of long-standing researchers like I am, particularly when it comes to the practical and clinical aspects of trial. So, my hope is really that we will really continue this as really an eye to eye and same level of partnership ultimately to improve quality of trials and ultimately, to improve the care and lives of our patients.
Phil Lake: Those are great thoughts to finish on so hopefully, they’ll be heard loud and clear by all of those who are in the industry. Okay, Kai, well, thank you very much for your time. It has been a pleasure speaking to you and I think we’ve got some very useful insights into respiratory development. Thank you very much.
Dr. Kai-Michael Beeh: Thank you, Phil. It’s been a big pleasure to talk to you and I wish you big success for that podcast and other parts of the series. Thank you very much.
Phil Lake: Thank you.
[Outro]: Special thanks to our featured guest Dr. B and to Phil for leading this week’s conversation on respiratory data capture. We’ve learned so much about capturing quality data and the many pitfalls that can disrupt results. Got questions? Let us know by emailing us at [email protected] We’d love to hear your feedback and answer your question on our next episode. As always, thank you so much for joining us. We’ll see you next time on the Trial Better podcast.
Dr. Kai-Michael Beeh, founder and Medical Director of insaf Respiratory Research Institute Wiesbaden, Germany.
Dr Beeh studied in Medicine and Art History in Frankfurt, Germany, he was clinically trained at Mainz University Hospital from 1997 to 2004. He performed basic research studies on lung inflammation at the National Heart and Lung Institute, Imperial College, London, UK in 2000, and received board certifications in Internal Medicine (2003) and Pulmonary Medicine (2004).
In 2004, he founded the insaf Respiratory Research Institute in Wiesbaden, where he also holds the position of Medical Director. After receiving the venia legendi for Internal Medicine from the medical faculty of the University of Mainz, Germany in 2004, he also serves as external lecturer (‘Privatdozent’; adjunct Professor) at Mainz University Hospital. Dr Beeh has designed and conducted more than 100 clinical trials in asthma, COPD and allergy, acting as Principal or Coordinating Investigator.
In 2009, he founded Aereon Consulting, a scientific service company, located in Wiesbaden, Germany. Dr. Beeh received a degree in healthcare “Market Access Management” from the European Business School, Oestrich Winkel, in 2015.
Dr. Beeh is the author of the popular international non-fiction book “Die atemberaubende Welt der Lunge” (The breathtaking World of the Lung) published by Random House in October 2018 (Germany, Austria, Suisse) and throughout 2019 in several countries globally. He is Editor of Pulmonary Therapy, Associate Editor of Advances in Therapy and has published or co-authored over 100 peer-reviewed articles on pathophysiology and pharmacotherapy of chronic airway diseases, including two publications in the New England Journal of Medicine. He has also edited and co-authored a textbook on COPD exacerbations in 2014, and reviews scientific articles for biomedical journals such as the European Respiratory Journal, Lancet Respiratory Medicine, and others. Dr. Beeh is a member of the German Society of Pneumology (DGP) and German Society for Market Access (DFGMA).
- Webinar: How to Position Oscillometry in Respiratory Clinical Research
- Webinar: Learn How to Apply Forced Oscillation Technique in Your Next Respiratory Trial
- Blog: Capture High Quality Respiratory Data: Don’t Let Spirometers Get in the Way
- Best Practices: Minimize Uncertainty and Risk in Your Clinical Trials