The World of Language Models in healthcare!

Date: January 13, 2023

Unlocking the Future: How Language Models are Revolutionizing Healthcare

This is an introductory post that will give you foundational concept of how LLM are built in context of healthcare in simplified terms. This will then help you to understand better claims and news about LLM and GPT in healthcare.

Large language models (LLM) are essentially trained to predict the next word in a sentence (basically an advanced autocomplete), but new capabilities (such the ability to summarize text and answer questions posed in natural language) become possible on their own without explicitly training for them as a feature of complex system, which allow the model to perform tasks such as pass medical licensing examinations, simplify radiology reports, extract drug names from a physician’s note, reply to patient questions, summarize medical dialogues, and write histories and physical assessments.

Section 1: What Makes LLMs Possible?

1. Learning on Its Own - Self-supervision

How is this possible. It starts with a simple idea of Self-supervision training, a learning approach in which machine learning model learns without relying on explicitly labeled data as examples. Instead, the model generates its own training objective from the input data without the need for human-annotated data, which can be time-consuming and expensive to produce..

A common type of self-supervision is in the form of an autoregressive training objective, in which the model is trained to predict the next word or token in a sequence, given the previous words or tokens. The training objective is to maximize the likelihood of the correct word given the context.

2. Building Language Skills - Transformers

Training in this manner is often the first stage in training LLMs (generative pre-trained transformer GPT) and helps the model learn language structure, grammar, and semantics. Transformer is a deep neural network architecture that is designed to be efficient at capturing relationships and dependencies between elements in a sequence, such as words in a sentence. 

Section 2: Training and Tuning LMs for Medical Applications

3. Getting Better at Specific Jobs - Tuning

Tuning refers to the process of adapting a pre-trained LLM (GPT) to perform well on a specific task or domain. This process involves training the model on a smaller labeled dataset that is specific to the target task. (During this process, the model’s weights and parameters are updated using pertinent examples to optimize its performance on the target task.)

The GPT-3 model was trained on 45 terabytes of text data comprising roughly 500 billion tokens (1 token is approximately 4 characters or three-fourths of a word for English text). In comparison GPT 4 was trained on 170 Trillion tokens.

4. Learning to Follow Instructions - Instruction Tuning

Instruction turning Refers to a kind of tuning in which training is done on a dataset containing pairs of instructions and corresponding desired outputs or responses.

Section 3: Medical LLMs in Action

5. Medical Language Models - Helping Doctors

Although general-purpose LLMs can perform many medically relevant tasks, they have not been exposed to medical records during self-supervised training and they are not specifically instruction tuned for any medical task. Medical records can be viewed as consisting of sequences of time-stamped clinical events represented by medical codes and textual documents, which can be the training data for a language model. Wornow et al8 reviewed the training data and the kind of self-supervision used by more than 80 medical language models and found 2 categories.

6. Medical Documents:

First, there are medical LLMs that are trained on documents. The self-supervision is via learning to predict the next word in a textual document, such as a progress note or a PubMed abstract, and conditioned on prior words seen. Therefore, these models are similar in their anatomy to general purpose LLMs (eg, GPT-3), but are trained on clinical or biomedical text.

researchers at the Center for Research on Foundation Models at Stanford University created a model called Alpaca with 4% as many parameters as OpenAI’s text-davinci-003, matching its performance at a cost of $600 to create.

7. Medical Codes:

Second, there are medical LLMs that are trained on the sequence of medical codes in a patient’s entire record that take time into account. Here, the self-supervision is in the form of learning the probability of the next day’s set of codes, or learning how much time elapses until a certain code is seen. As a result, the sequence and timing of medical events in a patient’s entire record is considered.

As a concrete example, given the code for “hypertension,” these models learn when a code for a stroke, myocardial infarction, or kidney failure is likely to occur. When provided with a patient’s medical record as input, such models will not output text but instead a machine understandable “representation” of that patient, referred to as an “embedding,” which is a fixed-length, high-dimensional vector representing the patient’s medical record. Such embeddings can be used in building models for predicting 30-day readmissions, long hospital lengths of stay, and in-patient mortality using less training data (as few as 100 examples). 

The World of Language Models in healthcare!
This website uses cookies to improve your experience. By using this website you agree to our Data Protection Policy.
Read more