Not everything needs an LLM: A framework for evaluating when AI makes sense

Question: What product should use machine learning (ML)?Project manager answer: Yes.

Jokes aside, the advent of generative AI has upended our understanding of what use cases lend themselves best to ML. Historically, we have always leveraged ML for repeatable, predictive patterns in customer experiences, but now, it’s possible to leverage a form of ML even without an entire training dataset.

Nonetheless, the answer to the question “What customer needs requires an AI solution?” still isn’t always “yes.” Large language models (LLMs) can still be prohibitively expensive for some, and as with all ML models, LLMs are not always accurate. There will always be use cases where leveraging an ML implementation is not the right path forward. How do we as AI project managers evaluate our customers’ needs for AI implementation?

The key considerations to help make this decision include:

1. The inputs and outputs required to fulfill your customer’s needs: An input is provided by the customer to your product and the output is provided by your product. So, for a Spotify ML-generated playlist (an output), inputs could include customer preferences, and ‘liked’ songs, artists and music genre.
2. Combinations of inputs and outputs: Customer needs can vary based on whether they want the same or different output for the same or different input. The more permutations and combinations we need to replicate for inputs and outputs, at scale, the more we need to turn to ML versus rule-based systems.
3. Patterns in inputs and outputs: Patterns in the required combinations of inputs or outputs help you decide what type of ML model you need to use for implementation. If there are patterns to the combinations of inputs and outputs (like reviewing customer anecdotes to derive a sentiment score), consider supervised or semi-supervised ML models over LLMs because they might be more cost-effective.
4. Cost and Precision: LLM calls are not always cheap at scale and the outputs are not always precise/exact, despite fine-tuning and prompt engineering. Sometimes, you are better off with supervised models for neural networks that can classify an input using a fixed set of labels, or even rules-based systems, instead of using an LLM.

I put together a quick table below, summarizing the considerations above, to help project managers evaluate their customer needs and determine whether an ML implementation seems like the right path forward.

Type of customer need	Example	ML Implementation (Yes/No/Depends)	Type of ML Implementation
Repetitive tasks where a customer needs the same output for the same input	Add my email across various forms online	No	Creating a rules-based system is more than sufficient to help you with your outputs
Repetitive tasks where a customer needs different outputs for the same input	The customer is in “discovery mode” and expects a new experience when they take the same action (such as signing into an account):— Generate a new artwork per click —StumbleUpon (remember that?) discovering a new corner of the internet through random search	Yes	–Image generation LLMs –Recommendation algorithms (collaborative filtering)
Repetitive tasks where a customer needs the same/similar output for different inputs	–Grading essays–Generating themes from customer feedback	Depends	If the number of input and output combinations are simple enough, a deterministic, rules-based system can still work for you. However, if you begin having multiple combinations of inputs and outputs because a rules-based system cannot scale effectively, consider leaning on:–Classifiers –Topic modellingBut only if there are patterns to these inputs. If there are no patterns at all, consider leveraging LLMs, but only for one-off scenarios (as LLMs are not as precise as supervised models).
Repetitive tasks where a customer needs different outputs for different inputs	–Answering customer support questions–Search	Yes	It’s rare to come across examples where you can provide different outputs for different inputs at scale without ML. There are just too many permutations for a rules-based implementation to scale effectively. Consider:–LLMs with retrieval-augmented generation (RAG) –Decision trees for products such as search
Non-repetitive tasks with different outputs	Review of a hotel/restaurant	Yes	Pre-LLMs, this type of scenario was tricky to accomplish without models that were trained for specific tasks, such as: –Recurrent neural networks (RNNs)–Long short-term memory networks (LSTMs) for predicting the next word LLMs are a great fit for this type of scenario.

The bottom line: Don’t use a lightsaber when a simple pair of scissors could do the trick. Evaluate your customer’s need using the matrix above, taking into account the costs of implementation and the precision of the output, to build accurate, cost-effective products at scale.

Sharanya Rao is a fintech group product manager. The views expressed in this article are those of the author and not necessarily those of their company or organization.