Webinar Materials

On September 26, 2023, Christina Robinson from the National Center for Biotechnology Information (NCBI) ClinicalTrials.gov team gave a brief overview of the modernization effort and how the modernized website better meets the needs of multiple user groups. Following the presentation, NLM product staff from the NCBI ClinicalTrials.gov team answered questions from the audience.

Links

• ClinicalTrials.gov

• Drive-thru Data: API

• Unified Medical Language System (UMLS)

• NLM Technical Bulletin

• NLM Support Center

Transcript

LOUISE: So welcome everyone to NLM Office Hours. My name is Louise To. I am a Technical Information Specialist from the Office of Engagement and Training at the National Library of Medicine (NLM), and I'm going to be your moderator for today's Office Hour. If this is your first time, NLM Office Hours is a chance to learn more about NLM products and services and a time to get your questions answered. To provide broader and continuing access to the information, these sessions are recorded and they will be posted onto NLM's website. So NLM Office Hours is also brought to you by the Network of the National Library of Medicine (NNLM), the education and outreach arm of the NLM. And the NNLM works to advance the health and wellbeing of everyone through improved health, information literacy, and information access. They work regionally, helping organizations within communities serve the public they know best, and also nationally, offering online educational programs. So visit NNLM.gov to learn more.  

Today, for NLM Office Hours, we have Christina Robinson from the ClinicalTrials.gov team. Christina will give a brief overview of the modernization effort and how the modernized website better meets the needs of multiple user groups. Assisting me in managing today's session is Michael Tahmasian and Kate Majewski from the Office of Engagement and Training at NLM. So you'll have three ways to communicate during today's session. You can use the chat feature and we'll be watching for your questions in chat and we'll relay them to our subject matter experts following the presentation. And you may also raise your hand if you'd like to unmute and ask questions verbally. I will be looking for those hands. And you can also use the feedback icons which is titled Reactions on the bottom of your Zoom menu as well. Also note that we have a closed captioner with us today. You can find the closed captioning button in your menu options. So oops, sorry-- Now without further ado, let me pass it on to Christina to tell us about ClinicalTrials.gov.  

CHRISTINA: Thank you so much, Louise and I will go ahead and share my screen. Thank you everyone for being here today. As Louise mentioned, my name is Christina Robinson, I'm product owner for the modernized ClinicalTrials.gov and I want to share a little bit with you about the modernization effort that we've been undergoing and what the new website looks like and how that hopefully meets all of your needs better than it has previously.  

So I want to share just a really quick overview of what ClinicalTrials.gov is for anyone who might not already be familiar. We are part of the National Library of Medicine. It is the largest registry in the world of clinical trial studies and results. And we have three primary user groups I want to just share here. We have sponsors and investigators. They submit data through the protocol registration and results system or PRS, that's going to be discussed separately in a different Office Hour session. And then we also have the patients and healthcare professionals and researchers. I'll speak about them in a little bit more detail later on, but they are our primary users for the public facing website.  

This graph is meant to give everyone an overview of the growth of ClinicalTrials.gov, which was first conceived as part of the FDAMA. That's the Food and Drug Administration Modernization Act that was passed in 1997. The website itself launched in 2000 and you can see quite a lot of growth in that time frame. I'll highlight just a couple of milestones along the way. In 2004, the International Committee of Medical Journal Editors, also called the ICMJE, introduced the uniform requirements, which means the minimum set of data that needs to be submitted in order for a study to be published in an academic journal. And they also required prospective registration or registration before the study actually began. You can see a little bit later in 2008 that's when the results database launched. So that was part of the Food and Drug Administration Amendments Act also called FDAAA and that expanded the types of studies that are required to be registered. It expanded the data that's meant to be registered with each of those studies and now summary results to be submitted for certain studies as well. And then further to the right, you can see we launched the modernization effort in 2019. At that time, we had a little over 300,000 study records and you can see now today we have over 460,000. So, we've added another 100,000 records just in the last few years as we're trying to modernize the website. So, it's, it's challenging to improve something and work on it while it continues to grow. So, it's been quite a challenging but also rewarding effort. And it gives you a sense of the scope of ClinicalTrials.gov and the kind of growth that we anticipate over the coming years.  

In terms of the modernization effort itself, we had three sort of overarching aims for this effort. And one of the big focuses, the focus that I'm going to probably speak the most about today, is improving the user experience. We also, of course, wanted to upgrade the technical infrastructure, but we have to do both of those things while we continue to support the existing legal, regulatory, and policy framework.  

This is the timeline for modernization. You can see that Year 1 was all about engagement. I'll talk a little bit more about that later. Year 2 was for development and that's when we launched the first version of the modernized website ClinicalTrials.gov Beta, it was called in the beginning. And that ran in parallel to the classic ClinicalTrials.gov. In Year 3, we launched the PRS Beta and we've continued making changes to both of those products. And then in Year 4, just this year, we launched the modernized ClinicalTrials.gov. The modernized website assumed the primary URL, ClinicalTrials.gov. The classic website remains available at classic.clinicaltrials.gov.  

So, I said I would come back to those stakeholders again. We have those three primary groups of external stakeholders. You can see patients and their advocates, that can include healthcare professionals, caregivers. And the majority of these, we know that when they come to ClinicalTrials.gov it's their first time on the website. We also know the vast majority of them want to join a study or want to find a study for someone that they care about to join and participate in. We have those data submitters who are giving us trial information. And then the data researchers, again, those can be those journal editors, landscape analysts, and other representatives of both academic and commercial organizations. And then we have internal stakeholders as well.  

One of the things that we learned early on in modernization is that a lot of our users struggle to understand a key feature of ClinicalTrials.gov and that is, that as part of the library, we make the study record data available. We provide access, just as any library provides access to books, we provide access to data, but we do not own that data. The sponsors who are funding and/or running a study are responsible for the data that appear on the website. But we do provide supporting materials and information wherever we can in plain language to aid understanding of the study record data.  

So in terms of engagement, I'm going to start on the top right-hand corner and go roughly clockwise and talk a little bit about all of the engagement that we've done over the last several years. You can see we began in 2020 with the public request for information and got over 1,000 actionable ideas for both the PRS and the public website. In the bottom right-hand corner, we had a dedicated Board of Regents Working Group and we've met with them 16 times over the last several years. We have representatives from each of those three external user groups. And so, their expertise has really been wonderful in developing the product and the website in particular. We've had a lot of great input from them and from their respective communities. You can see we've had blog posts, 5 webinars, 2 public meetings. In the bottom left-hand corner, we did over 100 formative interviews in the early days of the modernization effort, where we really dug deep with our users into both how they were using the existing systems and what they wanted the new systems to do for them and how that could make their work easier.  

And then finally, we've analyzed over 2,500 feedback comments submitted by users. This is crucial feedback. A lot of times, users have submitted email addresses and we've been able to reach out to those users for additional input. We've included them in usability testing, tree testing, card sorting, lots of activities along the way and it's really been a wonderful experience. And just, you know, a few more of the ways that we've really tried to make sure that the modernized website meets users' needs as best as it possibly can.  

In that formative research, we learned a few things. Well, we learned lots of things about our users, but I wanted to highlight a few of those here on the left-hand side. You'll see a representation of those patients and patient advocates- the caregivers. One of the things that's been really interesting, and also challenging, is that our user groups have different needs, but not only do they have different needs, those needs are often in direct conflict with each other. We know that they want search to be as easy as Google, and they want to understand more about the search results in the study records. Meanwhile, in that middle column, you'll see the data researchers represented. They wanted Advanced Search right from the home page. That wasn't the case previously, and they wanted to see as much data as possible in a small a space as possible. And then on the right-hand side, those data submitters again, they want to enter their study record data and move on to their next task.  

So now I'm going to run through the modernized website and just point out a few of the key features that we've made available that we think are really great for those users and just highlight some of those for all of you here. A couple of the things that I'm particularly proud of: we've built the entire website according to the USWDS principles. That's the US Web Design Service. So that means the website is in alignment with other government websites. We've built throughout the process for accessibility. We've also optimized for mobile throughout the process. And so, we think that that's going to work better for those patient users who really want to join a study.  

And let me just go through a few more of those things. So across the top here you'll see that we have a streamlined site menu. In particular, we've got information about the Application Programming Interface or API; that's really important for those data researchers. That was previously housed on a separate micro-site. The same thing for the PRS Info. So we want to make sure that all of the supporting information and materials is available in one place. Let me just click once more here.  

Across the top you'll see a brief disclaimer about the website, again, trying to communicate what ClinicalTrials.gov is and is not. Users told us that this is really important information. They want to see it front and center, but they don't want it to get in the way of actually searching for studies. So, there's additional information there under the accordion, or you can click to go to the full disclaimer.  

In terms of the Search function itself, of course, this is the primary feature of the website. We do still have the default search form, which includes Conditions and Other terms, but we added Intervention and Treatment based on that feedback from users. And this was true of both the patients and the data researchers that Intervention and Treatment is really one of the most important fields and so we've added that to that form. 

Location actually has been updated with the Google Maps API, so that is hopefully more accurate and there's a lot of flexibility here. You can see that on the Study Record whenever you visit in Contacts and Locations. There's some additional functionality there, but we think this works much better.  

Finally, under More Filters, you'll find all the rest of the fields that were previously available as part of Advanced Search are now available right from the home page. And the last thing I'll point out here is that on the bottom right-hand corner you've got that yellow Feedback button. That's where we get that feedback from users. That's downloaded regularly and analyzed and really has been crucial for continuing our work and reprioritizing as necessary.  

Next up on the Search Results page, I want to point out that you've got here on the left-hand side all of the same filters and fields that you had available from the home page, so you can make adjustments to your search right here very quickly and easily. In the middle of the page, you can see some more details about your search. We have an action bar where you can select studies and download them. And then you've got, in the middle of the page, there's one card for each of the studies in the search results. And this has key information to give users some indication of whether or not this is a study of interest for them.  

Next up in the top right-hand corner, you'll see that there's actually a secondary view, so there's Card View and Table View. And when you click on Table View, you're going to see a lot more studies and more detail about each of those studies. So, I'll just click on that now. So, this is what it looks like when you go into Table View. This is really important again, for those data researchers who want to see as many studies as possible and as much data as possible about each of those studies. If you look at the filters on the left-hand side, you can actually hide those to see even more of the data. I'll just do that now so you can see a little bit more of the data. Or if you want, you can click on Manage Columns. This is in that action bar. That'll give you a modal where you can add lots of other columns to the table, and even reorder them so that you're seeing the information that's of most importance to you.  

Next up I'll go into the Study Record, just highlight a few key features here. So we have an intro at the top of each of the study records. This is that, you know, again high-level information about each of the studies. We do have, now, tabs down below that, where you can go into different sections of the study. So if you want to see study details or if you want to see the Results or the Record History, you would go through the different tabs that are available in the Study Record. Finally, we have on the left-hand side, left-hand navigation. This is actually a feature that's available throughout the website. This menu floats with you as you scroll down the page, or you can use the links there to jump throughout the Study Record. And then across the top here, you can see we do have the option to download a single study record or expand or collapse all of the content available in the record.  

That's it for my short demo of the website itself. I want to just give you a couple more little pieces of information and then we'll move into the questions that we've been getting in the chat.  

So I mentioned previously that the modern website assumed that primary URL, ClinicalTrials.gov, in June of this year. You see that represented in the middle of the timeline with the green star. We have a few trailing features to finish up later this year and you can see that the classic ClinicalTrials.gov will be retired sometime in the middle of 2024 before we move into operations and maintenance.  

This is a quick overview of the things that we are currently working on. You can see ‘Manage Saved Studies’ and RSS right at the top of the list. We know these are very, very important features for many of our users. We're also working on the next steps to join a study. Again, those patients want-- caregivers, healthcare professionals, they're looking for a study that they or someone they know can join. We do have a few more pieces of functionality to add to the study Record History and the Search Results Table View as well as some content migration. And then we do know that Download and the API both need to have XML available and we need to rewrite our ingest before we can retire the classic website.  

Last but definitely not least, is this wonderful team of people who have been supporting the modernized website. Absolutely wonderful team. The website would not be where it is without all of them. And I can't tell you how much I appreciate each and every one of them. So with that, I will stop sharing and give it back to Louise, if you would.  

LOUISE: Thanks, Christina. Alright, let's go back to our slides and then-- OK. So now it's time to open up the floor for questions. We have several that have already come in, but the folks that are going to be helping to answer your questions include many of the subject matter experts from the ClinicalTrials.gov team including Rafis Ismagilov, Maureen Strange, Catherine Kihara and Kenneth Ni, in addition to Christina.  

And so, I'm going to first start us off with a question that came in in advance from Lauren Fletcher. And I actually saw Lauren in the audience. So, I'm happy to say that we have your questions cued up. The first question is, With the updates made to the site and export options, what is the best way to export results to Endnote and/or another reference manager? And I think Christina, you might be the best person to answer this one.  

CHRISTINA: Hi yes, I think that's probably true. So, we don't currently offer data in RIS format. We do know that there is a desire for the data in this particular format, and actually what would be really helpful for us to understand, is to get a sense of who's using it for those purposes and how they're doing so currently. So if you could just share either in the chat, of course you can also submit that through the Feedback button on the website, but would actually be really helpful for us to know how folks are doing that currently.  

LOUISE: Awesome. Thank you, Christina. So we also did receive a question about the PRS Beta, PRS standing for the Protocol Registration and Results System, which is where data submitters submit their clinical trial information to ClinicalTrials.gov. But I will say that we are having a separate Office Hour for the PRS Beta site. So if you are interested in asking questions about the PRS, please register for that session, which I did drop it in the chat, but we may need to drop it again. Michael, could you help me just redrop that link to the PRS session? That place would be the best time to ask questions about the PRS Beta because the team that works on it will be there, just ready to answer your questions then.  

OK, so moving on, let's ask the question from Helena. Will the new site eventually be able to give us the URL search query link for Download like the classic site does? And this is relevant to API, so I'm going to ask Rafis to answer this question.  

RAFIS: Let me show you my screen, just one second. So, there are many options to get the data. But if you are talking about API, you can play with it if you go to the menu Data About Studies, then Learn About the API. Then you can do a search here. It's similar to what we have in the front-end application. For example, you can specify Conditions or disease or Other terms here. For example, let's say ‘cancer’, and then you get one page of the data. If you need a URL, then you can take it here so it's URL. But be careful, it's only for one page. So now API is paginated, it's documented here on top. So you will have to get the data page by page. But if we are talking about getting all data like on the classic site, like for example here, Download, you can-- so I did a search for everything, so I just didn't enter any filters and I've got all studies. Then I want to Download and it says the maximum is 10,000 but I can download everything if I go through to this page and it explains that I have to use this link.  

So if the question about this link-- so currently we don't provide it as a link. It's part of our front-end application but it's doable the same on the modern site. So you just go, for example, home page, don't enter anything here, so all filters are optional. Click on Search, you will get all the results as well. Then you click on download JSON format to get all data. There will be XML as well. And then you get them as an array of studies or put every study into separate file and download them as zip archive. So then you click here and click download to get all the results and all data fields. Or you can specify your search, for example, ‘heart attack’. Then you'll get less results and still say Download and basically what to get. Maybe not all these filters you need, but some of them or any field you want to get. So that's all up to you. For example, Title and Organization of Point of Contact.  

LOUISE: Okay. Thanks, Rafis. I think related to this question, Susanna had asked if there were going to be future sessions focusing on API integration with NLM in general or specifically with ClinicalTrials.gov.  

And I will say with NLM in general, we have hosted a webinar in the past that sort of describes how you can use NLM APIs to access information quickly, so I'm going to drop the link to that webinar if you want to view it to get a good sense of the various products that do offer APIs.  

As far as ClinicalTrials.gov, I think Rafis did a good job of summarizing the steps for the API. I did see another quite lengthy comment. Let's see, about APIs. I think this one had to do with APIs. This is from Rob: It has been a few months but last time I tried I wasn't able to access the revision number for a study from the API. Because the different view URL requires the revision number, it is impossible for my software to create a link to the latest different view for example. Perhaps even more useful would be more detail on the revisions and dates via API. Okay. So that was more like some feedback from Rob. So thank you, Rob.  

All right then, so let's move on to the next question. Monica asks: With a label reflecting the study name-- Oh OK, so she was referring to the URL download link currently and copying a link for the new site to share with users. It simply says study record. ClinicalTrials.gov, whereas links from the classic site explicitly reflect the name of the trial. So if sharing multiple study links in a single message, it's cumbersome to distinguish between study hyperlinks. Okay. So I would say maybe Christina, if you have a response for this one.  

CHRISTINA: So, what I think Monica is referring to is the label that's in the browser tab. So, it would just say, you know, as she called out here, study record and then the URL, the primary URL which is just ClinicalTrials.gov. We do know that that is there is a desire for that tab title to include the study record title or some kind of identifying information related to the individual study record. So yes, we are aware of that. That is something that we do plan to update. We will be doing that at some point in the future.  

LOUISE: Great. Thank you, Christina. Next question is asked by Rob. Are there enhancements planned on the Search Result presentation? Today the order appears undefined. It can be controlled by filtering, but not by sorting or grouping by phase, date updated, etcetera. I'm going to pass it back to Rafis for this one.  

RAFIS: So in search results, in the front-end application, the results are always sorted by relevance to user-entered query for example here. But the API, it's actually available, so you can sort there. For example, you go to endpoint to get studies then you enter search, etcetera. And there is also parameter sort, which allows you to enter up to two fields. It accepts date fields, numeric field or relevance. So actually for those type of fields, it's possible to sort them. But on the grouping, yes, it's not available yet. I don't know if it's a priority, but we'll consider it. Thank you.  

LOUISE: Thanks, Rafis. OK, so the next question several people have asked and Christina quickly touched upon it in one of her last couple of slides, but the question is, when will RSS alerts for new information be active?  

CHRISTINA: So yes, actually really exciting news. Managed Saved Studies and RSS will both be part of our next release, which is actually coming quite soon. So keep your eyes peeled.  

LOUISE: Awesome. Thanks, Christina. Next question is from Anna. Are maps coming back? We are an information center that serves people with many different conditions in the US and all over the world. The maps really helped us find relevant trials.  

CHRISTINA: Hi, yes, thank you for that question. So we do know that the map feature actually is quite important for some of our users and we're actually exploring ways to make that available on the modern website. We're actually looking into that right now. Thanks, Louise.  

LOUISE: Great, Okay. Next question comes from Angela. Angela asks: I use ClinicalTrials.gov frequently for systematic reviews. In the older interface, you could eventually find a command line search box where you could copy, paste, and run your search. I haven't been able to find a search feature in the new interface. Is it available in the new interface?  

CHRISTINA: So I think Angela is referring to the expert search. It is not currently available. That is another feature that we're exploring the best way to carry that forward.  

LOUISE: Okay. Thanks, Christina. Question from Diane: Will the support materials, FAQ's and trainer material be on the new site?  

CHRISTINA: Yes, absolutely. We will be moving over that key content over the course of the rest of this year. So yes, you should actually see that in the coming months.  

LOUISE: Great. We have another question also from Angela: I'm also interested in seeing-- OK she-- I think you answered that. The expert search return, so I can enter complex systematic review search strings without having to use the advanced search boxes to break up my turn. So I think you you'd answered it, but eventually. Okay. Thank you, Angela. Mark that checked.  

We have some comments from Susanna. I have found that if you want to return the URL for any given search, the URL would take this format, Study URL. This is quite a long comment from Susanna and maybe I'll just ask Rafis, if you want to take a look at her comment in the chat and go into more detail there and explain it there, I'll let you do that. I'm going to move on to Rob's comments. Oh, okay. No, I've already addressed Rob's comments, so let's jump on over then to another set of questions. So oops, there we go.  

So is there-- or sorry, What kind of engagement have you done with patients and caregivers to inform the design of the new website? And I'm going to hand this question over to Catherine.  

CATHERINE: Yes, great question. We love our patients and I'm actually going to come on camera just for a second, because we would like to encourage you to continue giving us feedback via the Give Feedback comment card as Christina had said. Oh I'm not on camera, I'm sorry about that. But over the past, over the past year, we've conducted lots of usability testing session research activity with patient types, and we are encouraging patient caregivers to reach out to us through the Give Feedback comment card because we are interested in knowing what they have to say about the features. For example, for the Search feature we engaged quite a number of them. So far, we have about 30% of our participant pull from patients and patient advocates. So we would like to hear from you. Thank you, Louise.  

LOUISE: Yeah. And I've also noted that the ClinicalTrials.gov team has been working with the Children's Inn at NIH, which is a residence for patients and families that currently participate in NIH clinical trials. And so they've been collecting feedback from them about how their interactions with the new modernized site-- or they will be collecting new information about the modernized site soon.  

And then, actually I realized, Rob, you had a second comment that I didn't address. So, you said, “I understand I could build a different site on top of the API that does better grouping, sorting. I've done that with a prototype ClinicalTrials.gov site. Would prefer to have you do that for the main site. Thanks for all your work in this area. A concerned parent”. OK, so that was just some feedback from Rob, so thank you for making note of that.  

All right, let's move on to another question. Can we search for a trial after it's been completed? And I am going to ask if Maureen can answer that question.  

MAUREEN: Sure. Thanks, Louise. On the Search Results page in Focus Your Search, there are filters that indicate Study Statuses. And so if one were looking for only completed studies, they would tick the Completed studies filter and they would see in their Search Results the completed studies.  

LOUISE: Great. Thanks, Maureen. Let's have another question. Does CT do some sort of automatic term mapping similar to PubMed? For example, cancer maps to neoplasms. I'm going to ask Rafis, if you could answer this question.  

RAFIS: Yes, we use UMLS as the dictionary for the synonyms and they are shown in UI. Let me show it really quickly. For example if you go same, cancer, and it shows here, so that's what was used in search and that's it.  

LOUISE: OK. Thanks Rafis. And yes, Kate did clarify what UMLS stands for. It stands for Unified Medical Language System. OK, I'm looking at the list of questions and I think we've pretty much been able to get through all of them.