Member-only story
In most of my articles, I talk about how to use Azure resources to solve a business problem. For this article, I wanted to change tracks to talk about the lesser-known, rarely acknowledged facets of Data Engineering.
As the title states, you will never just be designing data pipelines….or may be you will if you are starting as an intern in this field. But, as you progress and mature, new responsibilities and tasks will converge in your direction and show up on your sprint board. Some of them will be unexpected and unplanned; nonetheless support will still be expected. In my experience, here are the 5 major extracurriculars that chip away most of the time of a data engineer:
1. Access Control
Organizations where the roles for Data Science, Business Intelligence and Data Engineering teams are clearly defined often maintain the DE team as the guardian of their data store. This means data engineers need to govern the access of business users and, at times, audit the access for the existing users.
What this means, in simpler words, is that data engineers would often find themselves handling a barrage of requests from various business users that need access to different datasets in different tables. Depending upon the business team, you’d need to decide whether a…
