eCeLLM Review: Instruction Tuning for E-Commerce LLMs

eCeLLM Paper Review: Instruction Tuning for E-Commerce (with Data Examples)

The era of LLMs has truly arrived.

Even in NLP research — which was once dominated by architecture-driven advances — it's now hard to find a paper that doesn't involve LLMs in some way.

Looking across the research landscape, the work can be roughly categorized as follows:

1. Getting the most out of existing LLMs (e.g., Prompt Engineering)
2. Specializing LLMs for specific tasks (e.g., LLM tuning methods)
3. Rethinking the core architecture of LLMs (e.g., SSMs, Mamba, Transformers)
4. Optimizing LLM performance (e.g., 1-bit LLMs)

These four buckets cover most of the space (though certainly not all of it).

The paper we're looking at today sits somewhere between categories 1 and 2.

On the "getting the most out of LLMs" side, approaches range from
simple prompt manipulation to tool chaining.

More sophisticated work, like GraphRAG, focuses on how the information itself is structured and retrieved.

On the tuning methodology side,

some work, like LoRA and QLoRA, focuses mechanically on the training method itself,

while other work, like Instruction Tuning, studies how manipulating the training data affects learning efficiency.

All of these approaches have demonstrated the ability to meaningfully improve LLM performance. The paper we're reviewing today continues in this tradition, with a specific focus on the E-Commerce domain.

In-Context Learning (ICL)

Instruction Tuning first appears in Google's paper "Finetuned Language Models are Zero-Shot Learners".

(Google again...) (As the title suggests, it's one step beyond OpenAI's "Language Models are Few-Shot Learners".)

The principle is straightforward.

Decoder-only Transformer models are trained using next-word prediction.

(More precisely, next-token prediction.)

Starting from the preceding tokens, the model repeatedly predicts the next token until the <EOS> token becomes the highest-probability output.

Models pretrained this way showed genuinely impressive NLP capabilities.

Think back to when Google released the Transformer and the BERT pretrained model.

Just a few years earlier, the field was largely dominated by word-lexicon-based approaches, with text embedding research just beginning to emerge.

BERT, given only a handful of examples — raw text, with no feature engineering — successfully performed tasks like prediction and classification.

Because these problems could be framed as traditional tasks expressible in numbers, even solving them this way represented a major leap forward at the time.

But as compute and model size scaled up, LLMs began surpassing traditional NLP tasks and approaching the level of general-purpose tasks that had previously required human intelligence. This sparked a natural question: can we further improve performance on specific problems?

This is the context in which OpenAI's Few-Shot Learning research emerged (linked above).

As I've noted in previous posts, LLMs exhibit dramatic capability jumps as they scale — and the same phenomenon was observed in In-Context Learning.

In-context information is the additional context included in a prompt. As models scale up, their ability to leverage this in-context information improves dramatically.

In other words, models develop a capability strong enough to be called genuine In-Context Learning.

Subsequent research continued to reveal just how powerful LLM In-Context Learning is — to the point where it became difficult to predict any upper bound on LLM capabilities.

Work, W. M. I. C. L. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?.

The prompt fed into a model is not additional information processed by some dedicated "special layer."
It's just text — the same as any other input.
Actually understanding context requires something like what's shown in the figure above.

And yet, simply by scaling up the model, LLMs were somehow doing exactly that.

Understanding the context of given data, understanding the format!

This wasn't just a function mapping inputs to outputs via tuned weights —

it looked like the model was actually performing something resembling genuine "comprehension."

Learning that doesn't require weight updates via "understanding" — this seemed completely impossible at the time.

The field took exactly one more step forward.

Pretraining had established that LLMs could acquire general-purpose capabilities and store them as weights. In-Context Learning had shown that LLMs could process and leverage context.

So what if you combined the two?
Can "context-processing ability" itself be stored?

Instruction Tuning

The answer was yes — this style of fine-tuning worked, and the era of Instruction Tuning began. Today it's harder to find a HuggingFace model without -instruct in its name than one with it.

Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., ... & Le, Q. V. (2021). Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.

The idea is simple.

You fine-tune a model, but the inputs and outputs consist of clear instructions paired with correct responses for specific tasks.

Unsurprisingly, performance on the trained tasks improves. But the more striking finding is that performance also improves significantly on unseen tasks the model was never trained on.

Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., ... & Le, Q. V. (2021). Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.

So how is Instruction Tuning actually done?

The key is selecting appropriate tasks and generating high-quality data for each one.

You need (instruction, input) → output pairs, where each output corresponds appropriately to its instruction and input.

The output format can be constrained — for example, choosing from a fixed set of options, or answering Yes/No — or it can be generated from one of several response templates.

Instruction-tuned with ECInstruct

Peng, B., Ling, X., Chen, Z., Sun, H., & Ning, X. (2024). eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data. arXiv preprint arXiv:2402.08831.

eCeLLM applies the Instruction Tuning methodology described above directly to the E-Commerce domain.

Similar attempts have been made before, but this work makes all data publicly available as open source.

The authors define a total of 10 tasks, grouped into 4 categories.

Several notable aspects of the training process are worth highlighting.

Multiple Instruction Settings

6 instructions are defined for each individual task.
The primary (major) instruction is human-authored;
the remaining 5 are AI-generated.

The major instruction plus 4 AI-generated instructions are used during training.
The remaining 1 instruction is held out as an unseen instruction for evaluation.

Data Pre-Processing

A few straightforward preprocessing steps are applied:

1. Remove duplicates between train and test splits
2. Enforce language consistency — English-only data
3. Strip non-English characters such as HTML tags and Unicode artifacts
4. When selecting data, include only products with detailed descriptions
5. Constrain text length to an appropriate range
6. Human manual review of the processed data

Parameters

eCeLLM fine-tunes large pretrained models using LoRA.
LoRA adapters are added to all projection layers and modeling heads.
Full hyperparameter details are reported in the paper; key settings are:

3 epochs, 5% warmup, lr 1e-4, batch size 128, LoRA alpha and rank 16.

Three size tiers of open-source models are trained — Large, Medium, and Small.
Large: models with 10B+ parameters (T5 13B, Llama-2 13B)
Medium: 5–10B models (Llama-2 7B, Mistral-7B)
Small: 3B models (T5 3B, Phi-2 3B)

Finally, let's look at a concrete data example to understand how training is structured.

An example data instance used for the AVE task (extracting an attribute value from product information).

Instruction:
Given the title, description, features, price, and brand of a product along with a set of target attributes, extract the value of each target attribute from the product information. Output the extracted values along with the source from which each value was extracted (e.g., title or features).

Input:
– Product title: Bencore Multi Functional Molle Tactical Messenger Bag.
– Product description: This sturdy and durable tactical shoulder bag provides a perfect and stylish solution for almost any situation. The bag is made of durable nylon that resists tearing and color fading. The bag features numerous MOLLE straps throughout for MOLLE accessories. As shown in the photos, it has ample storage space with multiple compartments and comes in a variety of stylish colors. This bag emphasizes design, comfort, and functionality — it is fully ergonomic and lightweight, with plenty of pockets and velcro patches throughout. The product comes with Bencore's lifetime warranty and satisfaction guarantee. Bencore is a leading manufacturer of outdoor apparel/accessories, ranging from paracord and backpacks to basic outdoor essentials.
– Product features: Durable, lightweight, heavy-duty nylon construction that won't tear or break in extreme conditions — lifetime warranty. Sturdy and spacious main drawstring-closed compartment provides a secure place to store your gear. MOLLE system compatible with most MOLLE accessories; front pocket for quick access; spacious inner pockets for separate storage. Zippered hidden rear pocket, fully ergonomic padded system, adjustable shoulder strap for comfortable handling.
– Product brand: Bencore
– Target attributes: Material

Output:
– Attribute: Material; Value: Nylon; Source: Product description.
– Attribute: Material; Value: Nylon; Source: Product feature.

The prompt used during inference after training is complete
(Zero-Shot Prompt, shown here with options; remove the options field if no options exist)

(For 1-Shot inference, add an example by directly providing the Input and Response)

System prompt: Below is an instruction that describes a task. Write a response that appropriately completes the request.

Instruction: {instruction}
– input: {input}
– options: {options}
– response:

Peng, B., Ling, X., Chen, Z., Sun, H., & Ning, X. (2024). eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data. arXiv preprint arXiv:2402.08831 .

Models trained this way show strong performance on the tasks they were trained on.

However, the key advantage of Instruction Tuning is that it should generalize to unseen tasks as well,

and this study only validates generalization to unseen instructions — it does not evaluate performance on tasks outside the ten presented.

Let's also take a closer look at the actual performance on the tasks covered in this work.

1. Does providing diverse instructions per task meaningfully help?

Providing diverse instructions does yield higher accuracy. This is a meaningful result, showing that using multiple instructions during Instruction Tuning is beneficial.

That said, this finding is not surprising — it is already noted in the Flan literature.

2. Is training a single model on all tasks simultaneously better than fine-tuning one task at a time?

Results are not consistent across all tasks.

It is also unclear how well the data for each individual case was matched, and the trend may differ as data volume increases.

Since training a single model on multiple tasks simultaneously is generally the standard approach, the practical takeaway is to identify cases where task-specific training yields higher performance and think about how to compensate for those gaps.

Data Examples from the Paper

Unseen instruction (not used during training)
Find and extract the values associated with the target attributes from the product information, and specify the source where each value was found (e.g., title, description, features, etc.).

Product description: This sturdy and durable tactical shoulder bag provides a perfect and stylish solution for almost any situation. The bag is made of durable nylon that resists tearing and color fading. The bag features numerous MOLLE straps throughout for MOLLE accessories. As shown in the photos, it has multiple compartments with ample storage space and comes in a variety of stylish colors. This bag is designed with an emphasis on design, comfort, and functionality — it is fully ergonomic and lightweight, with plenty of pockets and velcro patches throughout. The product comes with Bencore's lifetime warranty and satisfaction guarantee. Bencore is a leading manufacturer of outdoor apparel/accessories, ranging from paracord and backpacks to basic outdoor essentials.

Product features: Durable, lightweight, heavy-duty nylon construction that won't tear or break in extreme conditions — lifetime warranty. Sturdy and spacious main drawstring-closed compartment provides a secure place to store your gear. MOLLE system compatible with most MOLLE accessories. Front pocket for quick access; spacious inner pockets for separate, convenient storage. Zippered hidden rear pocket, fully ergonomic padded system, adjustable shoulder strap for comfortable handling.

Product brand: Bencore

Target attributes: Material

Unseen instruction (not used during training)
Analyze the titles of Product 1 and Product 2 and choose the option that best represents the relationship between the two products.

• Options:
    A. Users who viewed Product 1 are likely to purchase Product 2.
    B. Users who viewed Product 1 are likely to view Product 2.
    C. Product 1 is similar to Product 2.

Unseen instruction (not used during training)
Compare the title, description, manufacturer, and price of the two products to determine whether they refer to the same product, and output a simple "Yes" or "No."

Unseen instruction (not used during training)
Analyze the user's review and determine the sentiment according to the given options.

Options:
A. Very Positive
B. Positive
C. Neutral
D. Negative
E. Very Negative

Output:
A

###########################################################################################
Task 5. SEQUENTIAL RECOMMENDATION (SR)
Given a user's interaction history with products, predict the item the user is most likely to be interested in next.

Instruction:
Seed instruction: Based on the products a user has previously purchased, rank the items in the given options and predict the product the user is most likely to purchase next. Answer with one of the provided options.

GPT-4-generated instructions (generated to improve instruction tuning performance):
Generated Instruction 1
Based on the user's purchase history, rank the items in the options and predict the product the user will be interested in next.
Generated Instruction 2
Rank the items in the options and, by analyzing the user's previous purchases, predict the product the user will purchase next.
Generated Instruction 3
A user's purchase history reflects their preferences. Rank the items in the options according to the user's preferences and output the product the user is most likely to purchase next.
Generated Instruction 4
Based on the user's purchase history, rank the items in the given options and output the product the user is most likely to purchase next.

Unseen Instruction (not used during training) 
Based on the user's purchase history, infer the user's intent and predict the most likely product from the given options.

Input:
1st: M-Edge Latitude Kindle Jacket, Pink (Fits Kindle Keyboard). Electronics. Computers & Accessories. M-Edge.
2nd: Marware jurni Kindle Fire Case Cover, Black (will not fit HD or HDX models). Electronics. Computers & Accessories. Marware.
3rd: NETGEAR AC1600 Dual Band Wi-Fi Gigabit Router (R6250). Electronics. Computers & Accessories. NETGEAR.
4th: iMBAPrice 110014-1 (1-Pack) Gold Plated 2.4 GHz 3-Way Coaxial Cable Splitter F-Type Screw for Video Satellite Splitter/VCR/Cable Splitter/TV Splitter/Antenna Splitter/RG6 Splitter. Electronics. Accessories & Supplies...

Options: 
    A: T POWER 9v 12v (6.6ft Long Cable) Ac Dc Adapter Compatible with X Rocker Pro Series H3 51259 Video Gaming Chair 51231,51396 & V Rocker 5130301... 
    B: Boys Floatsafe Flotie Soft Fabric Armbands Floatie Blue For Kids Ages 1 To 3. Floatsafe Floatie 
    C: Anker iPhone Charger, Powerline Lightning Cable (3ft), MFi Certified for iPhone Xs/XS Max/XR/X 
    D: Curtain Drapery Rod w/brackets Small - Wrought Iron Hand Made. Home & Kitchen. Home Decor. Hand Crafted & American Made! 
    ... 
    T: Lorex ACCMIC1 Indoor Audio Microphone Accessory for Surveillance DVR's (Black). Electronics. Camera & Photo. Lorex

Output:
A

###########################################################################################
Task 6. MULTI-CLASS PRODUCT CLASSIFICATION (MPC)
Given a query and a product title, predict the degree of relevance between the product and the query.

Instruction:
Seed instruction: Analyze the relevance between the query and the product title, and select the appropriate answer from the given options.

GPT-4-generated instructions (generated to improve instruction tuning performance):
Generated Instruction 1
Analyze the query and product title to determine the relevance between them, and select the appropriate option from the provided choices.
Generated Instruction 2
Evaluate the relevance between the query and the product title, and select the most accurate option from the given choices.
Generated Instruction 3
Analyze the query and product title to assess the relevance between the two, and select the option that best describes it.
Generated Instruction 4
Determine the relevance between the given query and product title, and respond by selecting one of the provided options.

Unseen Instruction (not used during training) 
Compare the query and product title to determine whether the product fully satisfies the query specifications, and select the option that best describes its relevance.

Input:
Query: aj1 black and white
Product: Nike Men's Air Jordan 1 Low White/Gym Red, White/Gym Red/Black, 9

Options: 
    A: The product is relevant to the query and satisfies all query specifications. 
    B: The product is somewhat relevant. It does not satisfy some aspects of the query but can serve as a functional substitute. 
    C: The product does not satisfy the query but can be used alongside a product that exactly matches the query. 
    D: The product is not relevant to the query.

Output:
B

###########################################################################################
Task 7. PRODUCT SUBSTITUTE IDENTIFICATION (PSI)
Given a user's search query and a potentially related product, determine whether the product can serve as a substitute for the user's query.

Instruction:
Seed instruction: Analyze the query and product according to the given conditions to determine whether the product is somewhat relevant to the query. If it does not fully satisfy the query but can serve as a functional substitute, answer 'Yes'; otherwise, answer 'No'.

GPT-4-generated instructions (generated to improve instruction tuning performance):
Generated Instruction 1
If the product can serve as a substitute for the query, answer 'Yes'; otherwise, answer 'No'.
Generated Instruction 2
If the product is a suitable substitute for the query, answer 'Yes'; otherwise, answer 'No'.
Generated Instruction 3
Determine whether the product can serve as a substitute even if it does not fully satisfy the query's requirements. If so, answer 'Yes'; otherwise, answer 'No'.
Generated Instruction 4
Evaluate whether the product can function as a substitute even if it does not satisfy some of the query's requirements. Provide a binary answer of 'Yes' or 'No' based on this assessment.

Unseen Instruction (not used during training) 
Evaluate whether the product can serve as a substitute for the query, and answer 'Yes' or 'No'.

Input:
Query: fissler magic smooth-edge can opener
Product: KUKINO Manual Can Opener, Multifunction Handheld Food Grade Stainless Steel Can Openers, Black.

Output:
No

###########################################################################################
Task 8. QUERY-PRODUCT RANKING (QPR)
Given a user's search query and a list of potentially related products, rank the products by relevance to the query.

Instruction:
Seed instruction: Given a query and a list of products (A, B, C, …), evaluate each product title and rank the products in descending order of relevance to the query, with the most relevant product at the top.

GPT-4-generated instructions (generated to improve instruction tuning performance):
Generated Instruction 1
Evaluate each product title and analyze its relevance to the query, then sort the products in descending order with the most relevant product at the top.
Generated Instruction 2
Rank products A, B, C, etc. by relevance to the provided query, and output the sorted list with the most relevant product at the top.
Generated Instruction 3
Analyze the query and each product title, then sort the products in descending order of relevance to the query. Rank the most relevant product at the top and output the list.
Generated Instruction 4
Evaluate each product title's relevance to the query, sort in descending order, and rank the most relevant product at the top.

Unseen Instruction (not used during training) 
Compare the query against each product title to assess relevance, then output the products sorted in descending order with the most relevant product at the top of the list.

Input:
Query: high heel shoe chair
Product A: ORE International HBB1826 High Heel Shoe Display Jewelry Box, Cheetah Print.
Product B: Coconut Float Red High Heel Giant Pool Float (Adult Size), 91.
Product C: Wildkin Kids Wooden Bench Seat, Toy Box Bench Seat, Features Safety Hinge, Backrest, Two Handles, Dimensions 32 x 15.5 x 27 Inches (Wild Side) (LOD71001).

Output:
A, C, B

###########################################################################################
Task 9. ANSWERABILITY PREDICTION (AP)
Given a product-related question and reviews for that product, determine whether the question can be answered from the provided document.

Instruction:
Seed instruction: Given a question and a related document, evaluate whether the information in the document is sufficient to answer the question. Output "yes" if it can be answered, "no" otherwise.

GPT-4-generated instructions (generated to improve instruction tuning performance):
Generated Instruction 1
Assess whether the question is answerable; output "yes" if the document contains information that addresses the question, "no" otherwise.
Generated Instruction 2
Analyze the question and its corresponding document, then predict whether the document can answer the question. Output "yes" if the document provides relevant information, "no" otherwise.
Generated Instruction 3
Given a question and a related document, evaluate whether the document's information is sufficient to answer the question. Output "yes" if the document contains an answer, "no" otherwise.
Generated Instruction 4
Output "yes" if the given document can answer the question, "no" otherwise.

Unseen Instruction (not used during training) 
Predict whether the given document can answer the question and respond with "yes" or "no".

Input:
Question: Where do you purchase the paddles, or are they included?
Document: Very satisfying purchase and price! My son plays with this for hours. Easy to assemble and he loves it! Very happy with this purchase. There was a slightly awkward point during assembly — the instructions note that two people are needed. Really well built and sturdy. Great value. Fun and exercise the whole family can enjoy! Buy today and use it for many years to come. It's sturdy and well made, so it should last for years! What's great is that my son can play by himself ;)

Output:
No

###########################################################################################
Task 10. ANSWER GENERATION (AG)
Given a product-related question and a review that serves as a supporting document, generate an answer to the question.

Instruction:
Seed instruction: Given a question and a related document, generate an answer to the question based on the information in the document.

GPT-4-generated instructions (generated to improve instruction tuning performance):
Generated Instruction 1
Use the information contained in the document to generate an answer to the question.
Generated Instruction 2
Extract information from the supporting document to answer the given question.
Generated Instruction 3
Use the supporting document to answer the given question.
Generated Instruction 4
Extract information from the supporting document to generate an answer to the given question.

Unseen Instruction (not used during training) 
Use the information provided in the supporting document to generate an answer to the given question.

Input:
Question: Can you connect additional receivers to a single sensor? That is, can one sensor receive a signal and send it to two receivers?
Document: This unit works perfectly on my 1,200-foot driveway. The one missing feature is the option for different alert patterns when there are multiple sensors — for example, 1, 2, 3, or 4 beeps to indicate which zone the motion is coming from. If you want a reliable motion sensor at long range, this is the unit for you. I installed the system two weeks ago, 300 feet from the house to the end of the driveway, and it has not failed once. It worked fine even on rainy days with no false alerts. There's almost 40 feet left on the driveway, installed at the slightly sloped end. Large vehicles (e.g., garbage trucks, tractors, lawn mowers) moving slowly do trigger the alarm, but fast or small vehicles are not detected, which I really like. There are two receivers — one indoors and one in the back. This makes the alarm very useful. I've bought alarm systems from several brands, and at this price point I have no reason to be disappointed. This unit is about 200 feet from the receiver and works great. . .

Output:
Yes… but all receivers must have the same DIP switch settings.