11 IA’s Impact on the Robotic Operating Model The Robotic Operating Model (ROM) is what I call the management side of RPA. It distills Blue Prism’s 20+ years of automation experience and best practices, into a framework that can be applied to all types of firms. This framework’s purpose is to help ensure the continued success and growth of automation in a company. This includes areas that may not be immediately obvious, for example, securing executive sponsorship, developing career paths, choosing an appropriate organizational structure, and facilitating cultural adoption. Although the ROM isn’t a technology product, it’s still a key differentiator between BP and its competitors. Everyone who works in RPA should been familiar with the ROM, especially since its guidance is vendor-agnostic. As technologies and the regulatory environment change, so must our management frameworks. The ROM is constantly undergoing revisions and is currently on version 2.0. This new version of the ROM is organized into five foundations: Strategy, Workforce, Design, Development and Operations. Each foundation is further subdivided into six subtopics, leading to 30 overall topics. While IA is certainly mentioned in the ROM, it isn’t one of the primary focuses. As a framework, the ROM is meant to provide general guidance, leaving us to fill in specifics. Trying to fill in these specifics was one of the main goals of my IA research, and you’ll find many of my research results here. In this chapter, we’ll discuss how IA impacts the five ROM foundations, so that you’ll be better prepared to tackle IA implementation in your own organization: • Strategy • Workforce • Design • Development • Operations Strategy Of the subtopics within the Strategy foundation, three are more heavily affected by IA. These include the Future of Work Vision, Business Case and Value, and Governance, Risk, and Controls. The Future of Work Vision subtopic discusses the vision statement, mission statement, and objectives of the overall automation program. It also discusses the importance of having a communication plan in place to ensure that the IA vision is spread throughout the organization. Business Case and Value is about ensuring that IA is aligned with corporate strategy, and that there are measurable KPIs to support that argument. Governance, Risk, and Controls discusses the governance board and the management of risk. Future of Work Vision If we’re looking to transition from RPA to IA, it’s important that the head of automation and executive sponsors explicitly update the vision to include this. For the more specific portions of the vision (mission statement and objectives), we can specify whether focus should be placed on integrating pre-built ML services, such as API-based OCR, publicly available LLMs etc., or developing in-house expertise for building custom ML models. Business Case and Value Since ML is a trending technology, there’s no shortage of executive management that want to see it being used across their business. Despite executive support, we should still work out how IA can be measured as contributing to the overall corporate strategy. Commonly used KPIs, such as full-time employee (FTE) savings and hours returned to business rely on quantifying how much time is saved by the Digital Workers vs. a human worker. But for IA, the amount of time saved can be negligible, for instance, if we’re replacing an expert’s decision-making process that takes only seconds to perform. So, we may need to shift from time-based, and dollar savings-based KPI measurements, to business-value-based assessments. Examples of these can be found in the ROM 2 training materials. The total cost of ownership also changes significantly with IA. We still have the traditional RPA costs, but we also have the costs of ML on top of that. There are three primary scenarios that lead to different ML costing on an IA project. The first case is if ML development is outsourced to a third party. This will likely have fixed and variable development costs. There are also possible ongoing service costs if the model is hosted by a third party. If the model is deployed back on-premises, and managed completely by internal teams, there will still be internal ongoing costs. The second is consuming ML through an API service, which incurs per-transaction costs. These costs can be forecasted based on expected work volumes and the API pricing. The third scenario is if ML is developed in-house. Through cursory research, I found that deploying an in-house ML solution into production for the first time costs roughly $100,000 USD, excluding RPA costs. IA is similar to introducing any new technology, in that there are high costs for the first project, but the marginal costs decrease as more IA projects are deployed into production. Deciding to build ML in-house should be thought of as a multi-year endeavor, and not as a one-off or pilot. The costs of in-house ML are mostly from salaries and renting hardware. Outside of ML that’s completely consumed through API calls, models will have ongoing costs, as data and model performance need to be actively monitored, and models rebuilt. Governance, Risk, and Controls One way to think about IA governance is to take the governance concerns of RPA, and add them to the governance of ML. Just on its own, ML governance is a huge topic, so it’s imperative that the governance board has someone who is very familiar with productionizing ML and is able to keep up to date with the shifting regulatory environment surrounding it. A starting point for the IA team to develop proper governance is to look internally for existing data and AI policies around data privacy, data retention, and security. We also need to consider whether use of an ML model is advisable from a fairness and ethical perspective. Currently, very few companies have AI ethics policies in place. Some potential starting points for developing internal ethical guidelines are the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems, and the Montreal Declaration for responsible AI. Risks IA has numerous risks associated with it that aren’t present in RPA. First, we need to comply with existing and upcoming laws around AI. As discussed in Chapter 5, nations are already developing laws to govern the use of AI in business. Another category of risks are model-based, the primary one being low prediction accuracy. The performance of ML models is also known to degrade over time. Finally certain types of ML algorithms are prone to adversarial attacks. There are also data-based risks, which include data bias, data quality, and data drift. Data bias refers to data that has “undesirable properties”, for example when the collected data underrepresents certain populations when it shouldn’t. Data quality refers to the presence of “desirable properties”, such as having a sufficient number of samples, and having features that are highly relevant to the prediction that we’re trying to make. Data drift refers to changes in the underlying input data distributions, and it occurs naturally over time. An example of data drift can be seen in consumer purchasing behavior. Even among similar age and salary ranges, the types of items that people purchase today are different from what people purchased 10 years ago. The next class of risks that can be reduced through ML governance have to do with security and data access. The introduction of new servers that host models, and new staff that must interact with data and models, increases the potential areas of attack that can target an IA solution. Addressing risks through governance Governance can help address all of the risks that were mentioned in the previous section. For compliance, governance can mandate that all IA solutions have a way to disable ML predictions or a preference for algorithms that are inherently explainable. ML governance can help reduce model-based risks. Governance can require that we implement formalized review and testing processes to ensure that a minimum proportion of predictions are correct before allowing a model to reach production. We might also mandate that general ML explainability methods, such as LIME and SHAP are used to assess models. Governance should define requirements around the continuous monitoring of model performance, as models are known to lose predictive power over time. This includes the collection of human-reviewed predictions for regular review, and understanding the how the frequencies of predicted labels for classification problems change over time. Governance can address data risks by implementing standards for data, and training requirements for data scientists. An example to establish data quality might be to mandate that data must have a minimum of 1000 samples per label, and that each row must have fewer than 5% missing columns. Governance can request data scientists to complete training around identifying data biases before allowing them to work on IA projects. Governance can also set standards around the need to monitor data on a regular basis. First baseline statistics need to be gathered, usually from the training data. Some common statistics that are calculated include the mean, maximum, minimum, and standard deviations of data columns. Then, new statistics are regularly recalculated based on the input data used in production. Governance might mandate that this be done monthly, and that any changes in data columns with more than one standard deviation be manually assessed or trigger the model to be rebuilt and deployed into production. For security, we need to amend the existing governance policies to define who can make changes to these IA components and the procedures for doing so. Workforce We will dive deeper into subtopics Building Your Organizational Model, Adopting New Ways of Thinking & Working, and Roles and Career Paths. Building Your Organizational Model discusses different ways that the IA function can be structured in a company. Some examples include IA as a central unit that serves everyone, individual IA teams within different business units, or IA being completely outsourced to a vendor. Adopting New Ways of Thinking & Working is about influencing the organization and overcoming resistance to get buy-in into IA. Roles & Career Paths is about which skillsets and roles are needed to create successful IA outcomes, and career progression within the IA team. Building Your Organizational Model Some of the main things to consider regarding the organization model are where does ML expertise reside in the organization, how you engage with them, and whether ML expertise needs to be brought directly into the IA team. If the firm already has a centralized data science team, we have to consider their organizational model as well, and how they expect to provide ML expertise to the rest of the company. If the company is serious about IA, it’s expected that the automation team will have at least a few members who can perform data science work without relying on external teams. For the Center of Excellence (COE), Franchise, or Hub and Spoke organizational models, the central team should have in-house expertise that can productionize ML models. For Divisional, Divisional Alliance, and Outsourced Managed Service models, ML expertise can sit in those teams instead. Adopting New Ways of Thinking and Working It’s important to understand ways in which IA can be opposed by both individual staff, and management as the adoption of IA will likely be more difficult to manage than RPA, due to the widened scope of what can potentially be automated. During my research, I identified some important ways in which employees and management oppose IA adoption, and some ways to counteract this opposition. IA resistance from the employee-level IA can lead to a loss of job meaning, if work that’s deemed meaningful to the employee is automated. A real-life example of this is the replacement of social worker’s elderly benefit’s screening phone calls with chatbots and RPA. Interviews with the affected employees showed that they had lower job satisfaction post-automation. A questionnaire called the Work and Meaning Inventory (WAMI) can be used to establish whether employees have experienced a loss of job meaning after IA is implemented. Job meaningfulness can be thought of in four parts: the individual, the job, the organization, and society. If we think that IA will result in a reduction in job meaningfulness, we can try to counteract these by targeting improvements at these different levels. For instance, on an individual level, we could provide more flexibility and autonomy to affected staff by allowing work from home days. On the job level, we could change the significance, visibility and scope of the work of the affected people. On the organizational level, we could increase the number of corporate social responsibility activities. A more practical approach would be to ask affected staff whether the a proposed automation has the potential to reduce their job satisfaction and to avoid automating those areas. Another risk is reduced work preparedness. In one financial services firm, documents were digitized and automatically input into internal systems, instead of being input by the case workers themselves. Removing the manual entry tasks meant that less time was spent looking at customer’s details. The affected staff felt more anxiety and less prepared to directly interact with customers. Counteracting reduced work preparedness requires making access to data simpler or improving the presentation of the data through summary dashboards. IA potentially affects employees’ sense of overall job security. There’s two parts to measuring job security. First we can measure how “stable” an employee thinks their job is through a Job Security Index questionnaire. Next, we can measure the employee’s attitude, given how they view their own level of job security, through Job Security Satisfaction scale. If practical, we can engage HR to measure these before and after IA has been implemented, to determine whether there are job security concerns that need to be addressed. If the goal of IA isn’t to reduce headcount, make “no job loss” an explicit message and publicize which measurable metrics will be used to evaluate the success of the IA program. If the goal is to actually cut jobs, training plans should be prepared to educated staff that will be retained on how to work together with the digital workers. The job role definitions for staff should also be revised. These two actions can help reduce the perception of job insecurity. A key to counteracting the potential negative impact on employees more broadly is through developing an understanding of AI sentiment across the organization. In all companies, there will be people that support or want to work with AI technologies, and there will be those that do not. The first rollouts of IA should target individuals or departments that view AI positively, as this improves the odds of initial positive results, which can help convince other areas of the business of IA’s benefits. IA resistance from the management-level IA may face managerial resistance, as it’s often presented as a way to reduce headcount and costs. It’s natural for managers to be concerned about their budget, sphere of influence, and ability to meet KPIs, if their headcount is reduced. IA can lead managers to passively resist, stall, and even actively sabotage IA efforts. IA metrics that are gathered by uncooperative management teams will often be underreported to undermine IA. Studies have been performed to understand management resistance to automation. Senior management opposition is mostly due to a lack of knowledge. Middle management that directly supervise staff or processes that will be replaced by IA have more specific reasons for opposition. In order of importance, these are being unconvinced, not having enough training, and worrying about their individual job importance. The fear of losing headcount was also directly mentioned as a major concern of managers. The proposed ways of addressing managerial resistance in research are quite vague from a practical point of view. Suggestions to reduce resistance include education, using data to convince management, implementing change in a gradual manner, and training. One practical way of reducing the fear of headcount loss (assuming that it isn’t the primary goal of IA) is to explicitly ask managers to submit plans on what other value-added tasks staff will be performing post-IA implementation. Going through the motions of planning how their staff will be used can help alleviate fears. Managers also worry that IA can trigger employee turnover and lower employee retention rates. There have been studies specifically about AI adoption, and its impact of employees. As expected, staff with negative attitudes towards AI are more likely to quit if AI is adopted. Again, we should understand how staff perceive AI and target use cases that affect employees with positive AI perception. Turnover intention is weakened when employee’s feel they have support from the company. This includes many things, including team building exercises, career planning, employee development and education, etc. Another key management fear is that of financial loss. This has been expressed in two main ways. The first is financial loss through litigation, for instance, if someone belonging to a protected class feels they’ve been treated unfairly due to biases in an ML model. The second type of financial loss are from incorrect predictions, that lead to further incorrect processing. We can manage these fears by ensuring management that we have proper governance in place to review models, select use cases and review specific predictions. Roles and Careers Paths If the overall strategy of the organization is to consume ML through APIs or other pre-built services, there aren’t really any changes to the roles that are needed on the team. Existing senior developers or team leads should already be equipped to handle these types of integrations. The exception to this is needing ML reviewers, but this role likely sits outside of the IA team, on the business side. However, if the strategy is to develop ML capability internally, a few new roles emerge. The first is the IA data scientist. This person will be responsible for analyzing data, building models, testing models, and evaluating them. I’d recommend hiring a person with these skills or borrowing this person from elsewhere in the organization instead of developing someone internally as the first data scientist. The main reason for this is that the time investment needed to become a data scientist is extremely long. It’s also unrealistic to citizen developers This data scientist can then build up a team of IA developers, which is a hybrid role between data scientist and automation developer. IA developers will start off with a strong RPA base, and slowly build up their data science skills over time. IA developers should also keep up to date on what commercial ML services are available on the market. IA developers, with enough experience can become full-fledged data scientists as a lateral career move. An important question is who manages the ML infrastructure. As the IA initiative grows, we may need an IA technical architect to define the appropriate deployment method, manage deployments and rollbacks, monitor data, and other infrastructure concerns that support IA. Unlike the technical architect role which often isn’t a full-time position, the IA technical architect will likely become a full-time position as the IA program grows, due to the ongoing management needs of ML models. Design In this section, we will dive deeper into the subtopics of Assessment and Prioritization, which discusses how to select IA processes for development, and Requirements Design, which discusses capturing the process steps and functional requirements of the business process. Assessment and Prioritization As part of the assessment phase of process discovery, a use case should have the endorsement from an experienced data scientist as to whether the ML portion is even feasible or not, before moving on to process analysis. Part of the process analysis phase should include either the POC-style development of an ML model, or the testing of ready-to-use APIs, to ensure that there’s sufficient accuracy (or other desired metrics) to warrant continuing with the IA use case. Again, there must be someone on the governance board who is experienced with ML to green-light IA processes for development. One important area to assess is how IA will affect existing SLAs. While we’d expect IA to increase the overall throughput of work that can be completed, the SLA might also include quality-based targets, which would depend on the accuracy of the model that’s used. The introduction of new infrastructure to deploy ML is also a new source of downtime. Many ML models are cloud-hosted on the major cloud platforms, such as GCP, AWS and Azure. In December 2021, AWS had multiple outages during the workday. Azure also had a major outage in January 2023. If your model was hosted there at that time, there’s a chance that you’d have an SLA breach due to the ML model being unavailable. Requirements Design Under IA, we need to help the business elicit their criteria to trigger HITL review of predictions. As seen in earlier chapters, this can include random sampling, different threshold values for different labels, or formula-based methods. The specific threshold values themselves don’t need to be picked yet, as they should be chosen based off of experimentation on the candidate models. The business also needs to define how prediction data will be presented to reviewers, for example, through Excel, a custom developed website, or a database. Once a prediction is made, we need to know whether there’s a maximum allowable delay between a completed prediction and human review for SLA purposes. There can also be requirements around the data and model itself. In some use cases, some data fields, such as age and gender are protected, and cannot be used in the model. If data is deemed sensitive, it likely can’t be sent outside the organization, removing online ML APIs as a potential part of our solution design. If models need to be explainable, this implies that certain types of regression and tree models should be favored. We also might have desired algorithms already in mind, for example deep learning, or LLMs, which impose hardware requirements (GPUs) on solutions. We also need to understand whether predictions are time sensitive, as this also informs the data scientists on which algorithms are possible and roughly what hardware requirements are needed to deploy a solution. Development The Methodology & Teamwork subsection is affected by IA, as there’s a separate development cycle for ML models that runs in parallel to the traditional RPA development. In terms of Delivery Controls, having a high-enough quality model also often acts as a go-no-go decision towards continuing the development of the IA solution. Under IA, Testing & Quality Assurance never really stops as the ML model needs constant monitoring to ensure that quality levels are maintained. Methodology and Teamwork ML model development is normally independent from RPA development. While BP has a six-phase delivery methodology (Define, Design, Build, Test, UAT, and Deploy), this isn’t exactly applicable to ML model building. For example, many ML model development methodologies have data analysis and refinement phases, which aren’t needed in RPA. If the ML model is built by a different team, we don’t really need to think too much about what particular ML methodology is being used. However, if ML is being built internally, the IA team should look to standardize their ML model development approach. There are many examples of this online, for example, from AWS (https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/well-architected-machine-learning-lifecycle.html) and GCP (https://cloud.google.com/blog/products/ai-machine-learning/making-the-machine-the-machine-learning-lifecycle). It’s normal for the ML model development lifecycle to begin before the Process Definition Document is finalized, especially for gathering and analyzing training data. During the Design phase of development, we need to work with the data scientists to choose the interface between the ML model and BP, whether that be Web API calls, executables, programmatic scripts, or Code Stages. We also need the data scientists to provide different examples of prediction responses so that they can be mocked in BP. This allows RPA developers to test their work locally even if the ML program isn’t ready yet. The functional requirements should also be discussed, such as required SLAs, response times, whether we can send prediction requests in batch etc. This should all be documented in the Solution Design Document. It’s often the case that for the first few initial IA projects, the RPA team needs to borrow expertise from elsewhere in the organization. If the data scientists aren’t fully dedicated to the project, there’s a risk that their IPA work will be deprioritized and for delivery timelines to get pushed back. Delivery Controls During the Build phase of either RPA, or the equivalent ML development stage, we need to keep in mind that insufficient prediction accuracy can lead to a no-go decision in terms of continuing development on the overall IA solution. In a sense, the progress of ML model development should be slightly ahead of RPA development to minimize the risk of wasted development work. UAT of the ML model is likely to be done completely separate to RPA. Completion of ML model UAT should be part of the entrance criteria before starting the RPA UAT. Testing and Quality Assurance During the UAT of the ML model, it’s important to capture the predicted result and the confidence scores. This data is needed to help us calibrate the appropriate confidence scores needed for different labels, in case thresholding is used to determine whether human review is needed. One of the main differences between IA and RPA is that ML model QA never really stops. Data scientists should regularly be monitoring the results of ML predictions to detect potential drift. Operations IA affects Deploy and Release, as ML deployments are expected to be done even if there are no changes to the business logic (Process) or applications (Objects). The Support Model has numerous changes, as the addition of ML brings in numerous ongoing operations that need to be performed, including monitoring models, monitoring data, exporting HITL review results, etc. Deploy and Release As we saw in the previous chapter, ML models are expected to be deployed regularly into production, on a schedule that’s independent from Process and Object updates. The deployment methodology that’s used for the ML model informs the Control Room operators on how the overall IPA solution must be updated, to maintain auditability, and also how to rollback, in case there’s an issue with the newly deployed model. It’s recommended to choose a deployment method that doesn’t require downtime to rollback, and for the prediction response of the model to return the model version that was used to make the prediction, for auditing purposes. Deploying new ML models should also require a formal change request, which requires approval. Support Model IA brings significant additions to the support model. We now have to worry about the uptime of the ML model (business continuity), ensuring the accuracy of the ML model (this can be assessed through random sampling), and potentially the SLAs for human review. Each ML model will have their own ML reviewers, who potentially also need training and access to BP, as they might be manually changing the statuses of Work Queue Items or editing Session Variables to recreate review data. If the ML models are maintained within the IA team, the models and their data will need constant monitoring to prevent performance degradation and data drifts. This is usually done through web-based dashboards which read in ML server logs. This works if ML is deployed through Web APIs, but not if models are called through the CLI, or Code Stages. In this case, the support model needs to include exporting ML results, from the Work Queue or Session Logs, for data scientists for analysis and ingestion into those monitoring systems. Regardless of the method used by BP to make the ML prediction, we still need to feedback information regarding reviewed predictions to the data scientists, as that data isn’t present in ML API server logs. The monitoring of data and models can lead to regular updates and deployments of ML models. Other concerns, such as updates to the ML libraries, the addition of new training data received by the HITL reviews, or experimentation with new algorithms can also lead to model updates. Changes in business logic can also trigger changes to the ML model as well. The support team needs to be very familiar with deploying new models into production. Another major concern, (related to referral handling and exception handling although it isn’t directly one of the two) is something called time lag effects. Imagine that we’re a bank that’s using ML to determine whether an account opening application is fraudulent of not. If the ML model is flawed, many fraudulent account creation requests will have been processed further, with the data being sent into many other systems. Thousands of account opening applications might have been processed before the model flaw is detected. The amount of work that needs to be undone or corrected between making an incorrect ML prediction, and discovering it, is a time lag effect. Issues with time lag are inevitable, unless the ML model is perfect. The support model needs to consider how to deal with individual cases, for example, if a business user flags a case where an incorrect ML prediction has led to incorrect processing. The support model also needs to deal with batch cases, which can be caused by flawed ML models. It might even be necessary to create a separate automated process specifically to undo the steps that were performed by the IPA solution post-prediction. The support model should try to address how liability should be assigned to a prediction that has led to losses of some sort. At some point, an individual person, or a team will need to be held responsible for the incorrect processing caused by an incorrect ML prediction. Is it someone on the IA team or the data science team? Is it the business users who have signed off on the model testing results? Is it the reviewers (assuming that the prediction in question has been reviewed)? This is something that needs to be agreed on by all parties, before finger pointing happens. ML liability is an active area of legal research. By examining multiple papers on the subject, I found that experts believe that the company that makes use of the prediction will be held liable, even if the ML model development is completely outsourced. While the company that makes use of the prediction can try to seek legal damages against the third-party, it’s almost impossible to prove that the algorithm is defective. An intuitive assignment of liability would be to person or team that maintains the model, or has reviewed the prediction, but both of these can potentially be outsourced. If those functions have been outsourced, the next intuitive liability assignment would go to the business users that have signed off on the model after UAT. Finally, if the IA practice is advanced, it’s worth looking into implementing some ML interpretability methods, such as LIME and SHAP. While these interpretability algorithms are most often thought of as an auditing step prior to deploying a model into production, it will definitely find some use in helping the IA team investigate problematic predictions for the business users and find ways to tweak the model to avoid them in the future. Summary The BP ROM is a framework and methodology that guides the entire automation program. Having a good understanding of the ROM significantly improves the odds of reaching your automation goals faster and scaling your digital workforce more quickly. While the ROM is an invaluable resource, it doesn’t provide specific guidance on how to add ML predictions to your RPA processes. In this chapter, I went through the five ROM 2 foundations, and discussed the subtopics that are the most heavily affected by moving from RPA to IA. The discussion was informed by my IA research findings and my own personal experience in ML. IA mainly affects the ROM by adding in a separate set of ML concerns to the existing RPA ones. This concludes the section on the management aspects of BP. In the next section of the book, we’ll put everything together through the re-creation of two real-life use cases.