Case study
Enhancing Sweets Production with Advanced Forecasting Techniques
The project downline was to create an engine forecasting the demand for the sweets produced by the company and integrating this engine with the database and interface used within the company. The forecast was supposed to be computed on the basis of historical data, actions of concurrent sweets companies (promotions etc.) in correlation to calendar public holidays and events.
Technologies used:
.svg){kind=icon}
SCOPE OF WORK
The need for the new solution occurred when the company, using the MS Excel tools, was unable to further reduce the demand forecast error. The use of Excel was based on simply tracking the trends from the previous years and applying the trends to the actual data. It was a business bottleneck and further process improvements killer. Also, the need for more elegant tools occurred which would ensure integration with the IBM Workspace environment used within the company for reporting. A better forecasting system could improve production planning by adjusting output to align with demand fluctuations. This method would fine tune inventory levels to ensure product availability during demand times leading to increased sales. Aligning production, with market demand strategically can greatly improve the organizations results propelling it towards profitability and operational efficiency.
Summarizing, the main reasons for the decision of launching this project were:
- Using more comprehensive and accurate data forecasting algorithms than those offered by the MS Excel.
- Increasing the computation time of a particular forecast by dedicating a separate server for the forecast engine.
- Automatizing data source communication rather than picking the data from different Excel sheets manually.
- Parameterizing the forecast scenarios.
- Integrating all the data coming from different sources in one DB engine.
- Exploiting IBM TM1/Workspace layer for data presentation as an internal tool for all teams working within the company.
The goal of the project was to develop a tool enabling demand forecasting with the total error on the level up to 5%. The communication with the data sources was done through Online Transaction Processing (OLTP) and REST API. The project was integrated with the IBM Workspace and IBM TM1 environment.
Services we did for them:
- The forecast engine, fully integrated with the online IBM Workspace platform, which allows the users to exploit the engine features independently.
- The engine allows running multiple forecast versions with different parameters at the same time thanks to the parallel processing implemented.
- The documentation of all the data sources which take part in the solution.
Where it works:
Project development
01
Input data audit and implementing protocols for data sources communication
Recognition of input data, which involves identifying and acknowledging incoming data types and sources. Preparing the code for communication between these data sources and the main database. This process ensures efficient data transfer and storage. The final step involves developing a forecasting engine that utilizes algorithms and models to analyze input data.
02
Engine development kick-of
Start of implementation of the forecasting engine using data samples only as the input. By exclusively relying on data samples, the system can test and ensure the accuracy and reliability of the forecasting system.
03
Design of the input/output structures
Designing relation tables storing data from the source DBs and results from the forecast engine. The data indexing process aims to enhance retrieval speed by organizing and optimizing the data structure. This step facilitates quicker fetching of information, improving overall system efficiency. Implementing DB procedures automatizing the dataflow. By automating these processes, the system streamlines data handling, ensuring smooth operations and reducing manual intervention.
04
Engine development - testing version
The initial engine version was in development, focusing solely on historical data and excluding considerations for calendar events and concurrency actions. This phase involves creating the foundational version of the engine, which relies on past data patterns for analysis and prediction. By excluding calendar events and concurrency actions, the development process can prioritize and refine the core functionalities of the engine.
05
Calibration of the engine: 3 iterations
The team performed 3 iterations of calibrations of the forecasting model, changing the static parameters and observing the accuracy of the forecast. The parameter changes were done with the forecasting team and supervised by them. During these iterations, collaborative efforts were employed with the forecasting team, ensuring informed decision-making throughout the parameter adjustments. The process involved close supervision from the forecasting team, providing expertise and insights to guide the calibration process.
06
First production version of the engine development
Developing the first version of the engine which took into account all the input data and parameters, such as calendar events and market actions. This holistic approach allows for a comprehensive effort on every source of data, laying the groundwork of the engine that may incorporate every single factor.
07
The engine goes live
The engine is fully connected and communicated with the running interface and main DB as well as IBM TM1/Workspace environment.
08
Training of the forecast team
Training the forecast team how to pass parameters to the engine, how to prepare and select the input data, how to run the engine and where to expect the results. The team in the client company started using the created solution, completely replacing their old tools.
Challenges
The first insight of the developers was a need to split the team into two groups. Thanks to our efforts, we have made the Company aware that it was unable to provide the full documentation of the data sources from MS Excel and there was no data warehouse/data lake prepared for forecast engine source.
To resolve this, the first group was responsible for data sources audit, data integration and design of the input/output structures. The second team immediately started developing the forecast engine using data samples and then pouring in more data as the project developed. To keep the flow of information between the teams, we had daily morning calls deciding the next steps.
The most challenging issues were two things, namely data integration from different sources and calibrating the forecasting model. The first issue was solved by thorough audit of data structures present in the sources, creating a data flow diagram to comprehend the process. Calibration of the model was solved by three iterations of tests performed with the users team from the chocolate factory.
The data needed to perform the forecast were as follows:
- Historical data for each product and aggregated data for each set of products in the portfolio of the company.
- Data concerning the actions (promotions etc) of the concurring companies.
- Calendar events and bank holidays.
For each product or an aggregated set of products, the engine was processing anything from one year to 10 years of historical data plus other companies actions and calendar events. It was up to 20GB of data for the whole company portfolio products. This size of data allowed the engine to estimate the forecast with accuracy below 5%.
Insights
What was really demanding for me at first was a data audit to recognize structured but unstandardized and unintegrated data sources exploiting different technologies, which forced extra time allocation.
As for the Data Engineer that I am, the idea of using MS Excel for data forecast was outdated and inferred a bigger error than newer technologies offered by the developing team.
That is why integrating the data on the one platform, namely IBM TM1/Workspace, allowed a better information flow between the teams within the company. For example, the demand forecast plan was an input for the production planning team. So the more accuracy in the demand planning increased accuracy in the production, which in turn increased savings of the company
Project in numbers
2
years
6
developers
7
technologies
4
integrated computer programs
1
pretty neat forecast engine 🙂
Find out more
Check out our projects
From SMS chatbots to data visualisations, from data security to system architecture, we do it all, and we make it fun! More than getting the project done.
We care.
Financial Institution Associating Banks
Interhyp came up with a business need to verify their hypothesis about moving traditional letter communication into digital solution by creating a web application which will be served to their customers...
Got a project in mind?
Feel free to ask questions or see our handbook
