One of today’s hot buttons is AI. It doesn’t matter if you are a student, retiree, starting your career or holding a C-suite title, how to use AI (or how it is being used in daily transactions) is on everyone’s mind. For companies dipping their toes in the AI well, the question of Cost Optimization is a part of the discussion, too.
On September 23, 2024, Matthias Patzak with AWS, one of our partners, wrote an interesting blog post on the topic, which we would like to share with you, below.
“Matthias joined the Enterprise Strategist team in early 2023 after a stint as a Principal Advisor in AWS Solutions Architecture. In this role, Matthias works with executive teams on how the cloud can help to increase the speed of innovation, the efficiency of their IT, and the business value their technology generates from a people, process and technology perspective. Before joining AWS, Matthias was Vice President IT at AutoScout24 and Managing Director at Home Shopping Europe. In both companies he introduced lean-agile operational models at scale and led successful cloud transformations resulting in shorter delivery times, increased business value and higher company valuations.”
His Generative AI Cost Optimization Strategies post:
“As an executive exploring generative AI’s potential for your organization, you’re likely concerned about costs. Implementing AI isn’t just about picking a model and letting it run. It’s a complex ecosystem of decisions, each affecting the final price tag. This post will guide you through optimizing costs throughout the AI life cycle, from model selection and fine-tuning to data management and operations.
Model Selection
Wouldn’t it be great to have a lightning-fast, highly accurate AI model that costs pennies to run? Since this ideal scenario does not exist (yet), you must find the optimal model for each use case by balancing performance, accuracy, and cost.
Start by clearly defining your use case and its requirements. These questions will guide your model selection:
- Who is the user?
- What is the task?
- What level of accuracy do you need?
- How critical is rapid response time to the user?
- What input types will your model need to handle, and what output types are expected?
Next, experiment with different model sizes and types. Smaller, more specialized models may lack the broad knowledge base of their larger counterparts, but they can be highly effective—and more economical—for specific tasks.
Consider a multi-model approach for complex use cases. Not all tasks in a use case may require the same level of model complexity. Use different models for different steps to improve performance while reducing costs.
Fine-Tuning and Model Customization
Pretrained foundation models (FMs) are publicly available and can be used by any company, including your competitors. While powerful, they lack the specific knowledge and context of your business.
To gain a competitive advantage, you need to infuse these generic models with your organization’s unique knowledge and data. Doing so transforms an FM into a powerful, customized tool that understands your industry, speaks your company’s language, and leverages your proprietary information. Your choice to use retrieval-augmented generation (RAG), fine-tuning, or prompt engineering for this customization will affect your costs.
Retrieval-Augmented Generation
RAG pulls data from your organization’s data sources to enrich user prompts so they deliver more relevant and accurate responses. Imagine your AI being able to instantly reference your product catalog or company policies as it generates responses. RAG improves accuracy and relevance without extensive model retraining, balancing performance and cost efficiency.
Fine-Tuning
Fine-tuning means training an FM on a additional, specialized data from your organization. It requires significant computational resources, machine learning expertise, and carefully prepared data, making it more expensive to implement and maintain than RAG.
Fine-tuning excels when you need the model to perform especially well on specific tasks, consistently produce outputs in a particular format, or perform complex operations beyond simple information retrieval.
I recommend a phased approach. Start with less resource-intensive methods such as RAG and consider fine-tuning only when these methods fail to meet your needs. Set clear performance benchmarks and regularly evaluate the gains versus the resources invested.
Prompt Engineering
Prompts are the instructions given to AI applications. AI users such as designers, marketers, or software developers enter prompts to generate the output they want, such as pictures, text summaries or source code. Prompt engineering is the practice of crafting and refining these instructions to get the best possible results. Think of it as asking the right questions to get the best answers.
Good prompts can significantly reduce costs. Clear, specific instructions reduce the need for multiple back-and-forth interactions that can quickly add up in pay-per-query pricing models. They also lead to more accurate responses, reducing the need for costly, time-consuming human review. With prompts that provide more context and guidance, you can often use smaller, more cost-effective AI models.
Data Management
The data you use to customize generic FMs is also a significant cost driver. Many organizations fall into the trap of thinking that more data always leads to better AI performance. In reality, a smaller dataset of high-quality, relevant data often outperforms larger, noisier datasets.
Investing in robust data cleansing and curation processes can reduce the complexity—and cost—of customizing and maintaining AI models. Clean, well-organized data allows for more efficient fine-tuning and produces more accurate results from techniques like RAG. It lets you streamline the customization process, improve model performance, and ultimately lower the ongoing costs of your AI implementations.
Strong data governance practices can help increase the accuracy and cost-performance of your customized FM. It should include proper data organization, versioning, and lineage tracking. On the other hand, inconsistently labeled, outdated, or duplicate data can cause your AI to produce inaccurate or inconsistent results, slowing performance and increasing operational costs. Good governance helps ensure regulatory compliance, preventing costly legal issues down the road.
Operations
Controlling AI costs isn’t just about technology and data—it’s about how your organization operates.
Organizational Culture and Practices
Foster a culture of cost-consciousness and frugality around AI, and train your employees in cost-optimization techniques. Share case studies of successful cost-saving initiatives and reward innovative ideas that lead to significant cost savings. Most importantly, encourage a prove-the-value approach for AI initiatives. Regularly communicate the financial impact of AI to stakeholders.
Continuous learning about AI developments helps your team identify new cost-saving opportunities. Encourage your team to test various AI models or data preprocessing techniques to find the most cost-effective solutions.
FinOps for AI
FinOps, short for financial operations, is a practice that brings financial accountability to the variable spend model of cloud computing. For AI initiatives, it can help your organization more efficiently use and manage resources for training, customizing, fine-tuning, and running your AI models. (Resources include cloud computing power, data storage, API calls, and specialized hardware like GPUs.) FinOps helps you forecast costs more accurately, make data-driven decisions about AI spending, and optimize resource usage across the AI life cycle.
FinOps balances a centralized organizational and technical platform that applies the core FinOps principles of visibility, optimization, and governance with responsible and capble decentralized teams. Each team should “own” its AI costs—making informed decisions about model selection, continuously optimizing AI processes for cost efficiency, and justifying AI spending based on business value.
A centralized AI platform team supports these decentralized efforts with a set of FinOps tools and practices that includes dashboards for real-time cost tracking and allocation, enabling teams to closely monitor their AI spending. Anomaly detection allows you to quickly identify and address unexpected cost spikes. Benchmarking tools facilitate efficiency comparisons across teams and use cases, encouraging healthy competition and knowledge sharing.
Conclusion
As more use cases emerge and AI becomes ubiquitous across business functions, organizations will be challenged to scale their AI initiatives cost-effectively. They can lay the groundwork for long-term success by establishing robust cost optimization techniques that allow them to innovate freely while ensuring sustainable growth. Success depends on perfecting the delicate balance between experimentation, performance, accuracy, and cost.”
If you’d like to learn more about using AWS and AI, please feel free to contact us: 416.429.0796 or 1.877.238.9944 (Toll Free) or email us: [email protected].