LightGPT-instruct-6B Model

LightGPT-instruct is a language model based on GPT-J 6B. It was instruction fine-tuned on the high quality, Apache-2.0 licensed OIG-small-chip2 instruction dataset with ~200K training examples.

Model Details

• Developed by: AWS Contributors
• Model type: Transformer-based Language Model
• Language: English
• Finetuned from: EleutherAI / gpt-j-6b
• License: Apache 2.0
• Contact: Github issues

Prompting

The instruction template (adapted from the Alpaca README.md) is used to format the prompt:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:

Input prompt example:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
How can I tell if a pomegranate is ripe?

### Response:

The input ends with ### Response:\n to signal that the model should start generating the reply.

Example of generated continuation:

To determine whether or not a pomegranate is ripe, you should look for signs such as color and size. 
A ripe pomegranate will have deep red-orange skin with small seeds inside. You may also notice that 
it feels slightly softer than other fruits when squeezed. If you're unsure about ripeness, you can 
cut into one of the fruit to check its texture. The flesh should be soft but still yield slightly to 
gentle pressure. Finally, smell the fruit - if it has a sweet aroma, then it's likely ready to eat! 
Enjoy your delicious pomegranates!

Deployment

Please refer to Deploy models to Amazon SageMaker.

The example code below also shows how to deploy LightGPT-instruct to Amazon SageMaker

# pip install sagemaker==2.159.0
from sagemaker.djl_inference.model import DJLModel

#An AWS IAM role that has the permission to create SageMaker endpoint
sm_role = "arn:aws:iam::XXX" 

djl_model = DJLModel(
    "amazon/LightGPT",
    sm_role,
    dtype="fp16",
    task="text-generation",
    # number of gpus to partition the model across, default to use 1 GPU
    number_of_partitions=1
)

# this will take a few minutes to deploy
predictor = djl_model.deploy("ml.g5.2xlarge",
                             initial_instance_count=1) 

input_str = """Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
How can I tell if a pomegranate is ripe?

### Response:"""

data = {"inputs": input_str, 
        "parameters":
        {
            "max_new_tokens":400,
            "do_sample": True,
            "temperature": 0.7,
            "repetition_penalty": 1.1,
            "top_p": 0.8,
            "top_k": 50,
            "min_length": 200,
            
        }
       }

result = predictor.predict(data)

print(result[0]["generated_text"])

"""
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
How can I tell if a pomegranate is ripe?

### Response:
Ripe pomegranates are usually easy to spot, as they will be slightly soft and give when squeezed gently.
You may also notice that the skin of the fruit has begun to turn from green to yellow-green in color.
Additionally, you should smell the aroma coming from inside the fruit; it should have a sweet fruity scent.
Lastly, check for any blemishes or bruises on the outside of the fruit. If all these signs are present,
then your pomegranate is likely ready to be picked! Enjoy your fresh produce!
**Note:** To avoid bruising, make sure to cut the stem off before picking. Otherwise, you could end up
with a bruised and unappealing piece of fruit.  **Warning:** Be careful when handling and cutting
pomegranates, as they can easily bruise or break.
"""

Evaluation result

	LAMBADA PPL	LAMBADA Acc	Winogrande	Hellaswag	PIQA
GPT-J	3.99	69.7%	65.3%	66.1%	76.5%
LightGPT-instruct	4.33	65.0%	64.6%	63.9%	75.5%

Limitations

See limitations of GPT-J base model here.

The model may fail to follow instructions with long inputs (e.g. summarize a long text). The model often gives incorrect answers to math and reasoning questions.

Beware of hallucinations: Outputs are often factually wrong or misleading. Replies might look convincing (at first glance) while containing completely made up false statements.

This model is usable only for English conversations.