Amazon Q in QuickSight for FinOps Center Framework
Amazon Q in QuickSight for FinOps Center Framework
Generative-AI is going to have a significant impact on how customers operate their Cloud Estate. While the process automation is still being figured out, we looked to initially incorporate Generative-BI into our FinOps Center processes leveraging Amazon Q in QuickSight.
Big Idea
It has long been “Nirvana” to be able to have an assistant within a process application to reduce the “time on task” for the user/participant. The idea is to integrate current & accurate data to the application eliminating the need to navigate to another screen. With the Amazon Q in QuickSight for FinOps Center Framework we think we are providing an “aha” moment not only for the FinOps use case but likely also for how customers will be building their next generation of business process applications.
When we first conceived FinOps Center, we were 100% aligned to multi-account framework and allocated percentage of account bill to a financial budget. Working with Cost & Usage (CUR) date is overwhelming when you need to provide a business users experience within a web application. Given the Business personas that we build for and the complexity of data we did significant summarizing of data as we ingested CUR. Our rationale was the nature of the data was so complex that the details would be difficult for a non-technical or AWS SME to understand. Even with the good work of the CUDOS Dashboard & team the data is very dense for business users. With Generative-AI/BI our perspective had to change that CUR data can be consumable by the Business especially when integrated with the FinOps Center context.
Capability Goal
From an application feature perspective, we needed to think about the top-down Questions that users would ask about their spending data within their FinOps processes. Once we had Questions we need to look at which application tables would add value and context to the “raw data”. For the Business user experience, how would we integrate the Q&A bar into the application so that it was natural to use and further minimize time on task.
Technical Project
From a Technical Architecture/Component perspective, we had a bunch to figure out and lots of trial and error. All FinOps Centers tables are in DynamoDB which is a bit complex to integrated into the data pipelines to Amazon QuickSight's Datasets. While we have made other Architecture changes in Release 24.9.1 to store resource level spending in FinOps Center we decided that we would consume data directly from CUR in S3 via Athena so that Q could provide the most robust set of information for the user. For the FinOps Center data, we need to synch our data reloads with the updates from the CUR ultimately leveraging SPICE as the Data aggregation storage for the data. We talk more about this as the Q Refresh Automation pattern.
Figure 1: Data Aggregation of FinOps Center application Data and CUR to
Amazon Q in QuickSight FinOps Center Framework
We had the Architecture/Component strategy together of how this would all work but needed to execute. To be honest, we had a bit of running joke that we had to start over every third day. What we thought we could do in 1 Topic turned to 2 given limitation of how Datasets we could use in Named Entities or Calculated Fields. Pushing more Joins to the Datasets ended up creating more work in Athena to join data. All these Architecture strategy shifts only came after setting up a new Topic, adding Friendly Names, adding Synonyms, and asking Questions to evaluate Results. If, like we did, found that the results are not what we expected you need to really start from the view and work your way up to the Topic work. As the product matures this will likely be easier but plan more development time than what you think there will be and make sure you have alignment with the business around the Friendly names, Synonymous, and Named Entities.
User-Based Embedding of Q&A
FinOps Center leverages the Anonymous Embedding Pattern for Amazon QuickSight for the CUDOS dashboards but at time of this authoring Tag-Based Rules were not available for Amazon Q Embedding. All FinOps Center users are entered into a Cognito User-Pool. When a User is added to Cognito, a QuickSight Reader is created within QuickSight. Similar to how we enable a new QuickSight dashboards to be available to different Roles we added the same capability for QuickSight Topics. Each Topic has an ID that is added to the application to be available to targeted role for that Topic.
With the User-Based Embedding, we enable User-Based Rules based on the Users login information. Within FinOps Center, we know the Financial Budgets for user that is aligned to their AWS Spending Scope which we show them their content and processes. Where we landed was created a unique Dataset per Role and Corresponding Topic for that Role that is then Scope by their User-Based RLS.
As we evolve FinOps Center to manage things like AWS’s Migration Acceleration Program and Spending Recommendation from Cost Optimization Hub, we will create new Topics for Customers to implement for their users. Once the CloudFormation deployments are run in their environment, application administrators will add those specific Topic Ids to their environment. As illustrated in Figured 2, new Topics IDs just need to be added to configuration screen once installed.
On the User Experience, the Q&A Bar has been a bit of a tricky item to make function as we expected. FinOps Center is REACT application and we are rendering with the frame of our header. We wanted to be able to navigate to multiple Topics within the “Bar” at time of loading which required more depth for the Header. FinOps Center has both a Light and Dark mode which we can render as the experience changes. We had some issues with the Dark mode loading 2X the Bar on load which we resolved with some work with AWS.
Packaging and Amazon Q “BI-Ops”
FinOps Center is a 100% AWS Serverless application that is installed in the Delegated Admin Account of your AWS Cloud Estate. With that we need to be able to pack and deploy based on individual customers parameters of their FinOps Center deployment. We have created the CloudFormation scripts that create all the components of the Framework including the initial set of Topics, Friendly Names, Synonyms, Calculated Fields, and Named Entities. As we create more Topics and capabilities the Framework scripts need to be run in environment to update to the latest and greatest. Customers will be able to customize their topics as they see fit or create their own then add via the admin screens (Figure 2).
Data Refresh Automation
Operationally we are working with a lot of dynamic data. Given we are triggered from CUR updates we have several “Refreshes” that need to execute in sequence to provide updated data to the users. We call this Data Refresh Automation and included in the Amazon Q for QuickSight FinOps Center Framework.
- Receive and Update CUR
- Once CUR is processed, update Athena DB with FinOps Center context data
- Upon Athena Refresh, trigger a Spice Refresh
- Upon Refresh of Spice, Refresh Topic Data
How to Get Started
Our Amazon Q in QuickSight FinOps Center Framework is Free in the AWS Marketplace - https://aws.amazon.com/marketplace/pp/prodview-34e7u3qliurr4. You will need a version of FinOps Center prior to working with our team to install and configure in your environment.
Learnings with working with Amazon Q in QuickSight
For some level-set, we are not BI developers but “traditional builders” looking to provide additional capability.
- The Topic features were developed directly to Topics vs from Analysis which may have been a mistake. Since we were aggregating a number of different data in Athena we were running queries, updating Datasets, Creating Topics, viewing results, and then going back to the drawing board. No news but much of the work is backend data and queries.
- Like everything, you need to think about the experience of the users which in a process application means what Question would have at different stages. As mentioned previously we want through various of states of this which caused us to “start over every 3rd day”.
- Given the importance of RLS to our application and the after the fact learning of not having the anonymous embedding pattern available, that caused a bit more work on the application and user management. We are still working on how we can limit the Datasets and Topics due to our Role Hierarchy.
- If your use case has a similar requirement, a similar solution to our Topic Administration may be needed. As a Product Company we needed to implement the API BI-Ops which you will also needed for Multi-Account deployment.
- The Data Refresh Automation we've implement will need a similar solution in any implementation if close to real-time information is a requirement for uses.