12 October 2023 | 2 minutes of reading time
Thursday, October 12, 2023
As a non-technical person, I recently entered the world of data through i-spark. Running away screaming was an option, but I chose to dive into it instead. Learning new things is mostly fun! And what turned out? Many people do not have a clear picture of the steps within data management. So, the basics. I’ve mapped it out for you.
Data has to come from somewhere, we all understand that. Data sources contain information that, for example, has been collected on a website or from a CRM system, social media channels, or an ERP system. A Data Engineer is tasked with retrieving this data from the data sources and forwarding it to a storage place for raw data. This storage place could be, for example, a cloud data warehouse or cloud data lake. This part of data management is called ‘integration’. This can be done using various programming languages such as Python and Scala or via online tools such as Dataddo and Fivetran.
The retrieved data is currently a jumble of information that most people can’t make heads or tails of. So, raw data.
The next step is where the Data Analyst or Analytics Engineer comes into play. During the so-called ‘transformation’, he or she retrieves the raw data from storage and selects, combines, and transforms the necessary raw data into understandable, enriched data and sends it back to the cloud data warehouse or data lake. But to the specific ‘refined storage’ location. Imagine a physical warehouse where you also wouldn’t want goods from different suppliers to end up unorganized among the goods to be shipped. The Data Analyst or Analytics Engineer can do this with the help of the online tool DBT.
And then the last step of the data management process before concrete decisions can be made based on data. A Data Analyst or visualization specialist ensures that the necessary data is retrieved from the refined storage and ends up in a dashboard in an organized manner. For example, using tools like Looker or Klipfolio. It may also be that the Data Analyst or visualization specialist makes an analysis based on a specific customer question. Often, the enriched data is used to also feed other systems with information, such as an email package. All these systems and people who consume information from the storage are called ‘Data Consumers’. An overload of raw collected data is at this point in the process transformed into factual and reliable input for making forecasts and targeted decisions.
This way, the route your data must take before you can use it seems quite simple. But it’s not for nothing that Data Engineers, Analytics Engineers, and Data Analysts study for a few years. It remains a technical process with incredibly many possibilities that must be followed very precisely. At i-spark, fortunately, we have all these technically smart heads available under one roof and can take this process from A to Z off your hands. We have been doing this for many years, for large and small commercial companies. Want to know more? Feel free to contact us via email: firstname.lastname@example.org or call: +31(0)594 855888
Marketing- and Communicatie i-spark