AI technology is rapidly evolving, making it difficult to tie down the best practices and lifecycle flow for an AI development project.
The AI development team must grapple with new concepts, technologies and complex development tasks in order to select the right model, train it with data, and augment it with appropriate RAG processing, agents and prompts – whilst the business impatiently waits for miraculous results to complex and poorly specified problems.
As part of our research into the characteristics of information management for the AI era, we see a general pattern emerging that can help to bridge the gap between the business sponsors and the AI development team, shortening the development process whilst improving the chances of a successful outcome.
Introducing Data Readiness
The heart of the problem is data. How is the right type of data delivered to the AI development team to allow them to build the AI Application?
At the start of the project, the AI development team is given access to existing data that seems appropriate for the requested task. As the results of the early experiments emerge, new strategies must then be developed to acquire, transform and assure new data sources that will improve the results to a point where the AI Application delivers value to the organization.
How is this achieved?
Data Readiness is the term we use to describe the activity that focuses on fulfilling the data needs of the AI development team.
The heart of the Data Readiness activity is the Data Readiness Assessment. Here the team reviews the results and feedback of each iteration from the AI development team. In liaison with the business sponsors, they assess whether the current AI application can be deployed to the business. This deployment may involve integration into the existing operational systems.
If the current iteration of the AI application is not good enough, Data Readiness seeks to identify and acquire additional data that, it is hoped, will improve the AI Application. This may involve purchasing data, or collaborating with the various data engineering teams and data owners in the current organization. Extension to existing processes and systems may be requested to generate new data. Finally, Data Readiness negotiates the development of special services (called MCP tools) that give the AI Development Team access to the systems that the AI Application needs to call to acquire real-time data, and to take actions.
So the Data Readiness activity sits between the business sponsors, the AI development team and the data owning teams. It needs people with experience and knowledge of the business, with enough seniority to locate the right people and resources and to get them to take action. They may be a dedicated team, or a steering committee with representatives from each of the affected teams (including the AI development team.)
Activity flow between data readiness and other activities in an AI project
The Data Readiness activity provides the AI development team with the political power to make the resources they need available to them – and ensure their work is perfected to make it useful the business.