Resources:
- [email protected]: Use this email to let us know what you want to hear about.
In this special episode, we have Veeva’s team joining us for a round table discussion. We will be tackling some of the most common mistakes in randomization and trial supply management (RTSM), sharing proactive steps to avoid these pitfalls, and exploring real-life case studies with practical solutions.
Joining our round table today:
- Kelly O’Brien, RTSM Senior Project Manager at Veeva.
- Andrew Rohrbaugh, VP Strategy at Veeva RTSM
- Éric Hardy, Senior Director of Biometrics at Indero, previously known as Innovaderm
- Oliver Hewer, Project Manager at Indero
Common Mistakes in RTSM System Setup
Kelly O’Brien
One common mistake involves insufficient planning for conversations around user acceptance testing (UAT). This planning should include identifying participants in UAT, such as the sponsor, the clinical or biostatistics team. Understanding the participants helps frame the test cases for UAT. Knowing who will run UAT and making the test script comprehensive is crucial. The script should not only include a happy path, which tests expected functionality, but also negative checks. These checks are important for aspects like calculations and stratification, which are critical to the build of the RTSM system.
Andrew Rohrbaugh
Another important aspect is being proactive. Planning for UAT should begin early in collaboration with sponsors and CROs, considering the resources and timing involved. Waiting until the last minute to plan for UAT can lead to delays and challenges.
Éric Hardy
Allocating sufficient time for UAT on both the vendor and CRO sides is essential. UAT requires extensive documentation, and it is key to have clear documents since testing will be conducted against these documents. Both parties must clearly understand the expectations, needs, and requirements.
Oliver Hewer
Having a clear set of specifications upfront and thoroughly discussing them is critical. Ensuring accuracy from the beginning is essential. As Kelly mentioned, performing tests according to all necessary roles and anticipated situations during the trial is key. This approach helps avoid problems once the trial is underway. Although some issues may still arise, getting the specifications right and testing for all possible scenarios is key.
Real-life Examples
Andrew Rohrbaugh
Two key points come to mind. First, using RTSM beyond its intended purpose can be problematic. At its core, RTSM is for randomization and trial supply management. The guiding principle is that sites will use the system only if there is a compelling reason, such as enrolling or randomizing a patient or obtaining a kit number. Adding extra data points or tasks that belong in an electronic data capture (EDC) system can burden sites and lead to frustration. Sites are primarily focused on patient care and other visit-related tasks, so their priority is to use the system quickly and efficiently.
Éric Hardy
Ensuring clarity in all specifications is essential. If something is not clearly defined early in the process, it is possible to test the wrong thing or encounter unexpected system behavior. Additionally, trying to move too quickly can lead to errors. When everyone is rushing, it is possible to miss something, which can create more work later. Pushing timelines can have significant consequences, so these aspects are important to consider.
Andrew Rohrbaugh
This principle applies not only to UAT but also to the system build itself. A mentor once advised that just because a system can be built in 2 weeks does not mean it should be. Sacrificing quality for speed is never advisable whether during the build from the vendor side or during UAT.
Proactive Strategies for Teams to Implement
Kelly O’Brien
When discussing programming, it is especially important to scheme the randomization list and requirements. The biostatistician team needs sufficient evidence that the appropriate randomization scheme is being used and that the randomization list loaded in both UAT and production environments is as expected. To facilitate this, a specific role can be built in the RTSM, allowing the unblinded statistician to access the system during UAT and production to review these lists. Ensuring they match 100% to the provided list is imperative. Additionally, during UAT, the biostatistics team should have their own unique scripts that focus more on the randomization scheme rather than just the clinical flow, providing a more in-depth view of the randomization build.
Éric Hardy
Ensuring the biostatistics team can test all aspects related to randomization is indispensable. It is important to verify that every stratification has been covered to ensure correct randomization throughout the study.
Steps to Get Ahead
Éric Hardy
Preparation is crucial, and having a good kickoff meeting ensures everyone is aligned and understands the protocol and RTSM requirements. For repeated business, building standard documents and modifying them for specific studies, rather than using large, complex documents, has been effective. This approach makes it easier for everyone to understand and agree on the details.
Kelly O’Brien
During the kickoff meeting, the first opportunity arises to meet everyone involved in the trial. A good conversation to start with all stakeholders involves understanding each individual’s role in RTSM. The clinical team, inventory management team, biostatistics team, and sponsor team all have specific needs. Addressing these needs ensures the system is built to accommodate each user role. For example, if a clinical project manager needs access to turn cohorts on or off or manage caps, the system should allow for these changes. Continuing the discussion to define user permissions ensures the system is robust and tailored to each role, including blinding and unblinding access.
Andrew Rohrbaugh
Another aspect of being proactive involves not only building the system to meet protocol needs but also considering potential real-world scenarios. Protocols are hypotheses, and reality often differs. For instance, if there is a lower age limit but no upper age limit, the system may not account for this. It is important to consider whether there should be an age limit, or controls for dose levels or weights. Instead of focusing solely on assumed boundaries, the system should be flexible enough to handle unexpected situations. This approach prevents issues where patients are unable to complete transactions due to the overly restrictive RTSM designs based on assumptions rather than real scenarios.
Oliver Hewer
Referring to Kelly’s point about identifying user roles early on, this approach can significantly enhance UAT testing. It is beneficial to involve individuals from specific areas in UAT. For example, having a blinded statistician conduct UAT in their role or a CRA test their specific role adds valuable expertise. This method ensures the system is accurately set up to meet the needs and parameters of each role.
Andrew Rohrbaugh
Taking that thought further, involving sites in UAT can be highly beneficial. Some sponsors include key opinion leaders from specific therapeutic areas or studies in UAT. Their feedback is valuable for aligning site workflows with system workflows, ensuring a smoother overall process.
Éric Hardy
Planning ahead for integrations with systems like EDC or other relevant platforms ensures a seamless workflow. This proactive approach helps avoid potential issues and ensures that all systems work together efficiently, supporting the overall trial process.
Andrew Rohrbaugh
Integrations are indeed a critical area. Any system or component that interacts with the RTSM, such as those controlling dosing or eligibility, must be carefully considered. Technology is reliable until it fails, so it is essential to have proper error handling and backup methods in place. This ensures that any transmission issues, file failures, or problematic values can be quickly addressed, preventing interruptions and minimizing impact on sites and patients.
Thorough Planning and Clear Communication
Kelly O’Brien
For study development, starting with a kickoff meeting is essential. This meeting helps understand team roles and define stakeholders. Maintaining communication through weekly or biweekly meetings ensures ongoing discussion about the protocol and RTSM requirements. These meetings address team questions and ensure everyone understands the system being built. Sharing the system build during design reviews is beneficial. Regular communication and progress updates contribute to a successful build. By the time UAT begins, the team will have a clear understanding of the RTSM and what to expect from the system.
Andrew Rohrbaugh
A key aspect is making the best use of everyone’s time. Study startup is burdensome for both sponsors and CROs. Ensuring meetings are productive, progressing, and addressing questions allows focus and efficiency. This approach helps everyone move on to the next system, as multiple systems are often being set up simultaneously.
Another key element is clear communication around requirements. Translating the protocol into system-specific requirements must be precise. Ambiguity can lead to misinterpretation, resulting in an end product that differs from the original intent. Clear, unambiguous requirements are essential to avoid this issue.
Best Practices for System Validation and Training
Oliver Hewer
Andrew mentioned the burdensome nature of setting up these systems, which involves both protocol-specific and system-specific requirements. One best practice is to focus on user training and continuous monitoring. Instead of relying solely on off-the-shelf training materials, incorporate the insights gained from extensive testing and system setup. Include protocol-specific and system-specific requirements in the training materials and manuals. This approach ensures that users are well informed and can effectively use the systems during the trial.
Éric Hardy
Adding to the point about timelines, it is necessary to allocate sufficient time for UAT. In the CRO world, thorough documentation is essential. Every aspect of testing needs to be well documented, and this process requires adequate time to ensure completeness and accuracy.
Ensuring System Integrity and Reliability
Kelly O’Brien
The role of the unblinded biostatistician is fundamental for ensuring data integrity in randomization and kit lists, which form the backbone of RTSM. Allowing the unblinded biostatistician to review these important lists during system reviews is key for maintaining quality and data integrity.
Andrew Rohrbaugh
In both RTSM vendor system validation and user acceptance testing, it is essential to conduct appropriate testing, especially for complex or unique scenarios. While the “happy path” testing ensures transactions work perfectly under ideal conditions, it is important to perform boundary testing as well. This includes testing upper and lower limits, dosing tables, and any variable elements. Although not every detail needs to be tested during UAT, sufficient testing should be done to ensure confidence in the system’s build and functionality.
Case Studies
Andrew Rohrbaugh
A company reported errors in the randomization lists for several phase III studies, rendering the data invalid. Despite the drug being beneficial and outperforming the comparator, the data could not be utilized. This oversight led to the shelving of the compounds due to cost implications and resulted in laying off about 50% of the workforce. Consequently, a drug that worked will not be available for an extended period, impacting patients with the disease. This situation underscores the critical need for meticulous creation, verification, and testing of randomization lists within the vendor and during UAT to avoid substantial monetary and patient impacts.
Éric Hardy
It is indeed pivotal to ensure consistency and accuracy in randomization lists. I have encountered situations where discrepancies between lists used by different parties caused significant issues. Even something as simple as adding an extra column can lead to major problems, often only discovered at the end of the study. Such oversights can be devastating for a company, highlighting the importance of meticulous attention to detail throughout the process.
Andrew Rohrbaugh
There is an interesting aspect of human psychology in testing. Studies have shown that documents requiring fewer signatures are reviewed more thoroughly, as individuals assume others are reviewing when multiple signatures are required. This concept applies to RTSM and testing as well. By the time UAT is reached, vendors should have conducted sufficient validation. However, it remains essential to ensure the system meets protocol needs and everything is done appropriately. This is the core purpose of UAT.
Éric Hardy
Randomization is indeed a critical function of RTSM, though not its only use. Having an unblinded biostatistician periodically review the data ensures the randomization schema is respected. This oversight helps ensure the study proceeds smoothly and maintains data integrity.
Kelly O’Brien
The randomization specs outline the requirements and ensure that only the randomization biostatistician and the unblinded team review and sign off on them. This responsibility emphasizes the importance of the details in the list, such as the number of digits and the allocation of each stratification. These specs set the stage for quality and provide a reference for validation and testing.
Andrew Rohrbaugh
Absolutely, being proactive is essential. Having an unblinded statistician check the study at various milestones and time points is necessary. RTSM is a technology, and despite thorough planning, issues can still arise. Early identification of these issues allows for quicker adjustments, addressing them promptly and minimizing their impact. This approach helps ensure the study runs smoothly and maintains data integrity.
As we conclude another illuminating episode of Phase forward, we find ourselves at the crossroads of science and progress. Remember that behind the jargon in statistics lies stories of unwavering commitment, meticulous observation, and the pursuit of evidence that shapes our understanding of health and disease, stay at the forefront of knowledge and innovation, and follow. Phase forward on your preferred platform. My name is Valerie Coveney. Thank you for joining us. Until next time.
Let’s shape the future of research and make a difference in the industry, gain Indero’s support in your upcoming trial and propel your study to new heights.
