The report arrives on Thursday. They asked for it on Monday and, in the meantime, three people have spent the morning joining extracts in Excel. By the time it finally comes out, the decision has already been made with last week’s data.
That pattern has numbers behind it. At a telecommunications company, producing operational reports consumed 12.5 person-hours a day in low-value manual tasks and caused 63% of internal complaints from management (study by the Universidad Peruana de Ciencias Aplicadas). The cause was not the tools. It was fragmentation. Seven of those hours went on manually unifying data in Excel (the same study).
What you recover when the process stops depending on people
In that same case, an ETL script in Python with a centralised database and a Power BI visualisation layer brought the whole process down to 2.3 person-hours and freed the 3 people who were sustaining it (the same study). The 3 hours of building individual datasets and the 2.5 hours of report processing stopped being bottlenecks.
The savings vary a lot depending on the starting point. In five insurance campaigns at a financing company, dataset preparation times fell by 77%, 73%, 86%, 87% and 20% after redoing the workflows and changing the query tool (work on dataset preparation times in insurance campaigns). The campaign that improved least was the one that could not be migrated without deep changes.
Other teams start with frequency. An automation project with RPA in Python and Power BI took the indicators to an update every 30 minutes, with efficiency improvements above 90% and a 300% increase in update frequency (study by the Universidad Privada del Norte).
There is a less showy part that sustains all of the above. Standardising workflows, automating validations and maintaining traceability controls is what reduced rework and production incidents in the Big Data operation of a telecom operator (internship report at the Big Data Management of TIGO-UNE).
What to measure before automating anything
Before choosing a tool or setting up a pipeline, set aside a week to measure. These five numbers tell you where the real bottleneck is.
- Person-hours per report. Time the full cycle, from the request to the delivery. Not the script’s execution time, but what it consumes from your team.
- Hours of manual unification. If your people are pasting files into Excel, that is the first candidate. It is the most repeated cost.
- Internal complaints per report. A high rate usually points to a definition problem, not a data quality problem. Agree on what each metric means before automating it.
- Frequency that the business asks for. Daily, hourly or every 30 minutes completely change the architecture you need.
- Owner of each data item. Preparation, cleaning, quality and governance are phases with different owners, not a single department (data engineering methodology for companies by the Universidad EIA).
Where this approach fails
Automation does not fix a badly defined process. If two areas cannot agree on what an active customer is, the pipeline will deliver the wrong number faster and better looking. That part is solved with conversations, not with technology.
You also have to accept the initial cost. Documenting workflows, defining metrics, setting up test environments and validating before moving to production takes weeks during which you have not saved anything yet. In the case of the insurance campaigns, one of them only improved by 20% because its process did not fit the change (the same work).
And it is worth saying another limit out loud. If the information is scattered across several platforms and the processes are still manual, centralising it requires ordering the source before thinking about new models (analysis of queries and reports at a coffee chain).
At Data Innovation we build and operate intelligent systems for data and CRM from Barcelona, with more than 20 years of experience and a Human+AI approach. The technical part is the easy part. The hard part is getting business, data and operations into co-evolution without breaking what already works.
If your team is still spending whole mornings joining files and your management is deciding with last week’s data, we can help you order the process and measure what you gain with each change. Tell us how you work today on our contact page.