An AI project often starts in a back room, as a series of experiments. Its operation at that time can be low key and informal.
At the other end of the spectrum, an AI project that has blossomed into a major success story, typically required a change in the way the organization does business. Often this transformation occurs in the very business areas that supplied data and expertise during the AI Application’s development.
In earlier posts (first post on Data Readiness, second post on Data Readiness) , we covered how data readiness both bootstraps an AI development project and sustains it with additional data to improve the AI Application’s behaviour. An effective Data Readiness function is also key to enabling the business transformation that will accept the AI application into its operations. It is not that the Data Readiness function drives the business transformation. That is the responsibility of another team. However, it represents the ongoing data needs of the AI application to the business transformation team, helping them to make choices when designing the new organization’s operation.
AI can only interpolate using existing knowledge. Innovation and creativity, necessary to handle new situations, comes from people with expertise. For the future sustainability of the AI-enabled operation, it is critical to ensure that the equivalent data sources and human expertise that fed the development of the AI application remain part of the solution going forward.
The AI Development Team will rapidly iterate the AI Application. They can use tools such as MLFlow to keep records of which of their data sources were used for each iteration and the results that where achieved. However, these sources are often only copies of the original data source. Data Readiness must maintain the linkage between the data sources provisioned to the AI Development Team and their origin, in terms of systems, pipelines, business area, ownership, licences etc.
This traceability allows the teams to understand which data sources are critical – and may be even suggest new iterations of the AI Application to try that eliminates the data sources that are likely to be missing in the future, in order to understand the impact. The results may drive Data Readiness to seek different data sources, or may lead into alterations in the new business design.
As we have seen, the Data Readiness function is key to ensuring an AI application is fit for purpose, and the organization is capable of deploying and sustaining it. It involves openness, collaboration and good record keeping to keep all of the parties in sync. This takes time and focus. So where does the responsibility for Data Readiness belong in the team structure? For example, is it the responsibility of the AI development team? Or should the project sponsors manage this responsibility? Or does it belong with the data engineers that supply the original data sources, or should it be a separate team?
Each organization will have a different answer depending on how their skills are organized. What is important is that time is carved out for this activity and people with sufficient seniority and knowledge of the organization, its operation and data sources are assigned to the role.
This is blog 3 in the Data Readiness series.