Determining patient eligibility for complex clinical trials of advanced therapies is a notoriously difficult problem. It is the primary reason why so few patients use clinical trials and clinical trials are so difficult to fill (resulting in unnecessary deaths and stalled scientific innovation). Many attempts have been made to attack this problem: from software for  researchers to define trial eligibility more consistently to use of AI to try to assess patient eligibility. The problem remains unsolved.

Commercial clinical trial matching products still do not use patients’ labs, diagnostics and biomarkers to truly match to patients. Attempts to use fine-tuned biomedical Large Language Models to present them with patients and trials and ask if they match do not work (frequent hallucination in the face of complex reasoning is still an Achilles heel of LLMs). Other attempts to use LLMs in various phases of trial matching merely reduce the “screening time” (as an example, TrialGPT reduced the typically dozens of hours to be exhaustive of possibilities for patients by only 42%, according to their own tests). 

A Different Approach to Matching (EXACT)

By contrast, our goal with the EXtracting Attributes from Clinical Trials (EXACT) system is to create a fully automated system for finding matching trials, that allows doctors and patients to find matching trials in minutes. The system starts with a set of patient attributes, gathered from record import, patient entry of data, or by an onboarding interview with the patient. 

[Note: There are many systems now, both research efforts and commercial products, that can already convert unstructured doctor’s notes into structured form. There are standards such as FHIR HL7 that express the patient information as a set of standardized attributes. In the face of this, starting the matching process with an unstructured patient narrative (such as TrialGPT and other systems do) is an unnecessary level of ambiguity.]

It then shows the patient what trials they are eligible for and what trials they are just potentially eligible for, showing them what attributes can be added to get more eligible trials. Before any of this can happen, we need to get the clinical trials eligibility criteria in structured form from the unstructured prose that they are created in.

Structure from Chaos in Trial Criteria

Most clinical trials do not express eligibility in well-structured form with uniformly named attributes or even with all eligibility criteria expressed explicitly in the participation criteria. It has often been assumed that the researcher will use software to express such criteria, ensuring rigorous logical criteria that can then be easily used later for trial matching. However the amount of trials that have been created with such software is negligible, as an advanced search of clinicaltrials.gov for CTML (the leading effort in this space) easily shows.

A different approach is needed that works with the current dominant practice, unstructured trial text, and extracts clear criteria for patient attributes. The advent of Large Language Models (LLMs) makes this much easier as you can ask many questions (prompts) about the patient attribute criteria when presenting a trial to an LLM. Unfortunately, simply presenting the trial text to the LLM and asking questions such as “Is HER2 positive required?”,  “What is the minimum serum bilirubin level required?”, and “Are pregnant patients eligible?” will not in fact get you accurate values for all criteria. Trivial usage of ChatGPT supplied with some trials and some of your known attributes can establish that it just doesn’t work for this problem.  

LLMs do not support detailed and accurate interpretation of all criteria and will often “hallucinate” equivalencies that don’t exist. For example, there are two measures of patient “performance” or overall health: ECOG scores and Karnofsky scores, each on their own scale. Trials will usually use one or the other for eligibility, but the LLM will often add the other measure and use the wrong scale. Trials often require or exclude certain kinds of previous therapies, therapy components and therapy types. The patient records (or their self-assessments) will typically have just the named therapy, and what the therapy type is or the components of that therapy are not known by the LLM. There are numerous synonymous terms used in clinical treatment and assessment, and the LLM knows many but misses many and often misapplies a supposed synonymous patient attribute or treatment where it does not apply. Attributes are often expressed in varying units, which often do not match the units that are reflected in the patient record, and often the unit is left unspecified. With a clinical subject matter expert in the loop evaluating LLM responses, prompts can be massaged and expanded to drive them to correct extraction of attribute criteria.

LLMs will also often hallucinate or over-simplify complex eligibility criteria. As described in more detail later, the EXACT system transforms extracted logical participation criteria in Conjunctive Normal Form for easier validation by SMEs and easier determination of missing attributes for eligibility by doctors and patients. 

The Prompt Workbench

The EXACT system uses a Prompt Workbench (PW) which presents prompts for extracting trial eligibility attributes along with the values to a biomedical Subject Matter Expert (SME). CancerBot has several such SMEs driving the PW. The raw trial text is presented on the left.  For all attributes whether it is required or excluded is presented as questions to be sent to the LLM on the right. The current answer for each eligibility value is shown as “Required”, along with the LLM used to extract the value.  Each attribute also has the proper units that will be used to match patient records identified. Also  ground truth “labeled value” can be entered for a representative subsample of all trials to facilitate global evaluation of extraction accuracy.  

Each attribute starts with generic questions like “What is the minimum Karnofsky score required to participate?” and “What is the maximum Karnofsky score required to participate?” . The SME can then edit the prompt as needed to get good answers. In this example, it turns out that it is required to add “Do not use any ECOG Score reference” to the base question about Karnofsky score. The system has built-in knowledge of therapy hierarchies (therapies and their types and components), cytogenic markers for diseases, molecular markers for diseases and specific lists of common mutations for common genes. These are accessible to the SME/prompt engineer as built-in variables such as $therapy, $therapy_type, $therapy_component, $cytogenic_marker, $molecular_marker and $brca1_mutation. The system has knowledge of all the valid values for these built-in variables.  

The SME can then edit attribute prompts to questions such as “Is $therapy_type required for participation?” and the prompt extraction process will loop over all therapy types for all trials seeing which therapy types are required for participation.  Furthermore the matching engine will later use the built-in knowledge to match the type (such as “immunomodulatory drug”) against a specific patient’s therapy (such as “lenalidomide”).  

Once the SME has edited the prompt it will show up highlighted.  They can then click Test and the values for all edited prompts will be shown to them, along with the logic of why the LLM made the interpretations that it did. If the values are valid for all edited prompts the SME can click Save and the saved prompts will be shown in red as “ready for extraction” (prompts run against all trials).  If the values are deemed not to be valid further editing of the prompts may be necessary including potentially choosing another LLM.  

Attribute-Criteria Extraction Engine

Once the prompts are defined, on a periodic basis an extraction batch job is run against all clinical trials stored with all edited prompts by the Attribute-Criteria Extraction (ACE) Engine. Results of the extraction run are presented to the SME prompt engineers, especially for any extracted attribute criteria values that are different from known to be correct “ground truth labeled values”. To respond to those problems prompt engineers can then edit the prompts further or choose different LLMs to extract the attributes in question. 

The ACE engine pipeline has a complex set of ETL processes that act before the LLM-based attribute criteria extraction is loaded. This allows the LLMs to be focused on only the attribute criteria which need it. It also ensures that the LLM works off of pre-processed data that will give the chosen LLM the best chance of extracting attribute criteria successfully. 

Figure 1 - ACE Engine Architecture

Eligible/Potential Matching Trials UI

Now that we have trials with structured information, we can use our partial set of patient attributes to determine a set of eligible and “potential” matches. To do this most effectively, the EXACT system expresses trial eligibility in Conjunctive Normal Form when presenting eligibility criteria to doctors and patients. 

[Note:The most prominent effort to define trials more rigorously is Clinical Trial Markup Language which allows eligibility rules to be defined by researchers with any arbitrary boolean expression. The performance of matching against thousands of trials with arbitrarily expressed complex logical criteria is much lower. More importantly doctors or patients trying to understand whether they match the trial cannot interpret the arbitrarily complex logic for eligibility in those trials.]

Here is an example of complex criteria for clinical trial eligibility:

Inclusion Criteria:

• Patients with HER2-positive 

• OR Patients triple-negative breast cancer)

• Must have PIK3CA mutation 

• OR Must have PTEN loss

• ECOG performance status of 0-1 AND age ≥ 18 years.

• No prior treatment with AKT inhibitors OR if treated, must have had a treatment-free interval of ≥ 6 months.

Exclusion Criteria:

• Active brain metastases AND concurrent corticosteroid use.

• Pregnant OR breastfeeding.

It is not easy to scale matching of such expressions over thousands of similarly expressed trials, which is part of the reason that commercial clinical matching products don’t do this level of detail of matching (they stick to a handful of attributes like disease, stage and grade), leaving it to the trial researcher or some kind of “human in the loop” consultant to perform the true match manually. 

More importantly, patient records are almost always incomplete. Not all patient health attributes are known, and it is work to get all necessary attributes for a trial match to a known state. With such complex expressions it is quite difficult to call out which attributes are missing to get trials from potentially eligible to eligible. 

Conjunctive Normal Form For Simpler UI

Expressing this logic in Conjunctive Normal Form (CNF), a set of AND conditions at the top level and OR conditions at lower levels, allows for rapid search by the matching engine. But more importantly it allows bringing in the patient or clinician into the matching process by presenting “partial matches” in an easy to understand way, and where the gaps in patient information are even clearer.   

Using DeMorgan’s law, let’s present the complex logical expression above in CNF:  

(HER2-positive OR PIK3CA mutation)

AND (ECOG 0-1)

AND (age ≥ 18)

AND (No prior AKT inhibitors OR treatment-free ≥ 6 months)

AND (NOT brain mets OR NOT corticosteroids)

AND (NOT pregnant)

AND (NOT breastfeeding)

When a patient or doctor is trying to determine if the patient matches the trial by viewing the criteria and patient attributes in a user interface, the “laundry list of ANDs” can make it straightforward. Let’s imagine that we know all of the patient attributes mentioned but don’t know whether they have a PIK3CA mutation, whether or not they have loss of function of the PTEN gene, whether they are triple negative and whether or not they have brain metastases.  We can then present the seemingly complex CNF criteria as the following list:

Note that while they have other unknowns even referenced in the logical expression, for the purposes of matching all they have to do is get a genetic test for either the PIK3CA mutation or the PTEN loss.  Then their eligibility can be determined. We did not need the answer to brain metastases or whether they are triple negative because the expression short-circuited at the first part of the OR clause. 

In the original “simpler” non-CNF form what tests can be performed to determine the match is not nearly as clear. The patient or doctor can then see the missing elements that turn a “potential match” into an “eligible match”. A bot interface (or other user interface mechanism) can guide the patient or doctor to get just the right test to complete the patient’s eligibility. The tests are performed and the missing questions are answered by the doctor or patient and the patient’s eligibility status is ascertained.  

Patient Attribute-Trial Criteria Harmonization (PATCH) Engine

The PATCH matching engine takes all of the currently known patient attributes and finds both Eligible trials and Potentially Eligible trials. “Harmonization” reflects driving to a standard vocabulary for patient attributes and the criteria values used therein (other matching engines such as MatchMiner use the term harmonization here). PATCH orders the trials by the number of missing attributes and reports back to the bot and the user in the UI the attribute that will move the most Potential trials to Eligible status. 

PATCH also has built-in knowledge of several concepts such as the hierarchy of therapies, components and types so that it can do inexact but logical equivalence. For example, a trial might require a history of immunomodulatories (iMiDs). Based on its knowledge of therapies, components and types, PATCH knows that if the patient has taken VRD in any line of therapy that the R refers to Revlimid, which is an iMid. PATCH also knows the categories of previously existing conditions. It also has knowledge of certain computed criteria such as MeetsCRAB or MeetsSLiM which are named OR criteria of more fundamental criteria (CRAB is calcium elevation OR renal insufficiency OR anemia OR bone lesions). It even has knowledge of geography so patients can find local trials by entering their location. 

System Architecture 

Below is a simplified higher-level diagram that represents how the entire system operates. The system assumes:

• a corpus of trials for a set of target diseases imported via API from clinicaltrials.gov and other trial databases

• a Prompt Workbench (PW) which allows refining of prompts to LLMs to extract accurate criteria from the clinical trials’ unstructured text

• an Attribute Criteria Extraction (ACE) engine which executes the defined prompts against all trials to get attributes for all trials

• patient record info with a partial set of health attributes, which may or may not be sufficient to determine trial eligibility

• the Patient Attribute-Trial Criteria Harmonizing (PATCH) matching engine which uses the CNF-expressed trial eligibility criteria to rapidly determine the eligible trials, ineligible trials and potential trials for the patient (ones that are missing some information)

• a human participation user interface, optionally driven by a chatbot, which shows the set of definitely eligible trials, definitely ineligible trials and potentially eligible trials, directing the patient or doctor to the minimal set of tests or questions necessary to determine eligibility. 

Figure 2 - EXACT System Architecture