Do you know the basic steps of the data management process?

12 October 2023 | 2 minutes of reading time

Integration

Data has to come from somewhere, we all understand that. Data sources contain information that, for example, has been collected on a website or from a CRM system, social media channels, or an ERP system. A Data Engineer is tasked with retrieving this data from the data sources and forwarding it to a storage place for raw data. This storage place could be, for example, a cloud data warehouse or cloud data lake. This part of data management is called ‘integration’. This can be done using various programming languages such as Python and Scala or via online tools such as Dataddo and Fivetran.

Storage & Transformation

The retrieved data is currently a jumble of information that most people can’t make heads or tails of. So, raw data.
The next step is where the Data Analyst or Analytics Engineer comes into play. During the so-called ‘transformation’, he or she retrieves the raw data from storage and selects, combines, and transforms the necessary raw data into understandable, enriched data and sends it back to the cloud data warehouse or data lake. But to the specific ‘refined storage’ location. Imagine a physical warehouse where you also wouldn’t want goods from different suppliers to end up unorganized among the goods to be shipped. The Data Analyst or Analytics Engineer can do this with the help of the online tool dbt Cloud.

Data consumer

And then the last step of the data management process before concrete decisions can be made based on data. A Data Analyst or visualization specialist ensures that the necessary data is retrieved from the refined storage and ends up in a dashboard in an organized manner. For example, using tools like Looker or Klipfolio. It may also be that the Data Analyst or visualization specialist makes an analysis based on a specific customer question. Often, the enriched data is used to also feed other systems with information, such as an email package. All these systems and people who consume information from the storage are called ‘Data Consumers’. An overload of raw collected data is at this point in the process transformed into factual and reliable input for making forecasts and targeted decisions.

Interested in having your data process managed by i-spark?

This way, the route your data must take before you can use it seems quite simple. But it’s not for nothing that Data Engineers, Analytics Engineers, and Data Analysts study for a few years. It remains a technical process with incredibly many possibilities that must be followed very precisely. At i-spark, fortunately, we have all these technically smart heads available under one roof and can take this process from A to Z off your hands. We have been doing this for many years, for large and small commercial companies. Want to know more? Feel free to contact us via email: hallo@i-spark.nl or call: +31(0)594 855888 

Esther Dijkstra,
Marketing- and Communicatie i-spark