Figure 1

Multimodal data sources and opportunities applied in medica field. The large-scale multimodal models should be able to handle as many types of data that doctors can access in their clinical work as possible, even those that some doctors cannot access, such as some new disease data or some unpopular diseases. Unifying these data as the input of the large-scale multimodal model for processing, including data from medical sensors, genetic groups, proteomics, and so on. It can allow the AI model to fully consider multi-perspectives as much as possible, providing a multi-dimensional understanding of the disease, helping to make more accurate recommendations or decisions, and more effectively reduce doctors’ burden in daily clinical work [4]."

Figure 2

GPT-1 Model Structure. Perform the pre-training through the Transformer structure on the left, and then fine-tune through the linear and softmax layer structures according to different types of downstream tasks on the right. One disadvantage of this structure is that all downstream tasks require fine-tuning, which increases a lot of work during the use of the model [13]."

Figure 3

GPT-3 using few-shot learning for English to French translation. In-context learning is a concept introduced by GPT-3, which leads to the idea of prompt learning. Rather than changing parameters for different tasks, the model can use the inputs and outputs of the tasks as prompts, and generate predictions based on the test set inputs. The user guides the GPT-3 model to learn this “thinking style” by giving questions and answers related to the questioning style to the GPT-3 model, so that when solving similar problems in real situations, they can start to answer questions along the user desired thinking path [16]."

Figure 4

Generating medical reports using ChatGPT. Due to its outstanding ability to summarize data information and extract key points, it can produce a high-quality medical report by feeding basic patient information into it, once it has learned the standard format of the report [17]."

Figure 5

Using the method of chain-of-thought to guide the model to generate correct answers. Due to the use of few-shot learning method in this case, the model imitates the processing in the similar questioning and logical example given above and transforms the new and complex problems into each step of simple but basic according to the chain-of-thought. This method enables the model to learn how to logically decompose complex problems into simple steps and tackle them respectively. The advantage of doing so is to improve the generalization ability of the model in tasks that require strong logic, such as solving mathematical problems [20]."

Figure 6

Two-stage prompt process demonstration. GPT-3 can produce better solutions to mathematical problems when prompted with “Let's think step by step”, showing the ability of chain-of-thought. This method can help the model generate more accurate results when facing complex problems and limited sample sizes situations [20]."

Figure 7

InstructGPT Training Process. The training process of InstrumentGPT consists of three steps [2]. They used a PPO (Proximal Policy Optimization) [22] network to produce outputs. The reward feedback model gives feedback to the PPO network. These steps were repeated until the InstrumentGPT model aligns with human values [2]."

Figure 8

The volume and distribution of the data were trained by InstructGPT. InstructGPT useed a small amount of labeled data by human for training, but it shows a high robustness that aligns with human values, making this method more feasible for medical scenarios with less labor cost [2]."

Figure 9

Comparison of pretraining fine tuning, prompting, and instruction tuning. Prompting and instruction tuning differ in how they activate the language models' abilities. Prompting triggers the completion ability of the models by needing few samples to fine-tune them for a downstream task. However, instruction tuning triggers the understanding ability of the language models and enables them to learn from the multiple downstream tasks. The model can follow more explicit instructions from users without fine-tuning on few samples, which improves their generalization ability across various tasks [23]."

Figure 10

Schematic diagram of pre-training and fine-tuning stages of BioGPT. First, It splits biomedical text into words and turns each word into a word embedding vector for the input layer. Then it passes them through multi-layer one-way Transformer network to get a context sensitive hidden layer representation for each word. In the pre-training stage, the model parameters are optimized using the autoregressive language model objective function. The model predicts the next word by giving a text sequence. In the fine-tuning stage, a specific output layer is added depending on the downstream tasks, and the model parameters are fine-tuned by the task related data and objective functions [27]."

Figure 11

Schematic diagram of working strategy of PaLM-E and applicable into downstream tasks. PaLM-E can use visual data to improve its language processing skills, create language that explains images, and give instructions to control robots to do complex tasks. PaLM-E is a decoder only LLM that can produce text completion in an autoregressive way with a given prefix or prompt. It uses multimodal statements that have visual, continuous state estimation, text and so on as its training data [28]."

Figure 12

The stratrgies and mainly structure of ChatCAD, which leverages the medical domain knowledge combining the logical reasoning skills of LLMs and the computer vision comprehension competence of existing medical image CAD model to assist doctors having a all-round understanding of symptoms and improve diagnostic process along with better therapeutic effectiveness [32]."