My golden framework to distinguish the nice, the dangerous, and the evil engineers in all fields, together with knowledge
To engineer is to design or construct one thing utilizing scientific ideas— Cambridge Dictionary.
All of us love good engineers, they construct unbelievable bridges, roads, rockets, functions, and knowledge constructions that make our lives simpler and pleasant each day.
By the identical logic, dangerous engineers is not going to make lives a lot better. If we rent them, they’ll design and construct one thing, however take extra of our time, cash, and power.
However do you additionally know, that outdoors the spectrum of fine and dangerous, there are additionally evil engineers, whose mindset is to not construct, however to not construct.
As an engineer myself, and somebody who has labored with a number of engineering groups sporting the product proprietor/ undertaking supervisor hats, the totality of my experiences has advised me one thing in regards to the good, the dangerous, and the evil engineers. I really like good engineers, I’ve empathy for dangerous engineers, and I despise evil engineers.
By the tip of this submit, I’ll inform you basically, what are the variations between such varieties of engineers. However first, let the story be advised from a extra anecdotal perspective.
Reflecting again by yourself experiences and information of the engineering world, what do you suppose are the widespread behaviors of the nice, dangerous, and evil engineers?
Under are my observations:
Good engineers:
- They acknowledge the issues
- They remedy the issues utilizing a sustainable method
- Additionally they remedy different issues associated to the recognized root trigger
Unhealthy engineers:
- They acknowledge the issues
- They remedy the issue for a short-term
- They create extra issues by fixing the unique drawback
Evil engineers:
- They faux to not see the issues
Let me make it simpler to think about these three engineer persona by describing a concrete instance within the knowledge engineering world.
Take an information engineer constructing a pipeline, which copies a set of uncooked knowledge tables from a transactional knowledge warehouse right into a container within the cloud. Following the medallion structure, the place knowledge goes by way of the bronze, silver and gold layers, they first clear the info and dump them right into a set of bronze-layer tables within the designated knowledge lakehouse. Subsequent, they normalize the desk within the silver layer, in addition to set up relationships between them. Lastly, they be part of a number of tables collectively in a view and create new options to signify the enterprise metrics to be fed into Tableau dashboards.
Throughout the testing of the dashboards, it’s famous that there are lacking values for a sure column in some information. Enterprise customers are involved as they see greater than 50% of information with lacking knowledge for that column, however in addition they acknowledge that the info could also be incomplete on the supply. Now, the engineers might want to examine and resolve the issue.
A great engineer will:
- First, they know very nicely how that column acquired reworked from bronze to gold to silver layer. In different phrases, they’ll know the precise knowledge lineage of the column with lacking knowledge.
- Determine a pattern document with lacking knowledge within the gold layer, however has knowledge on the supply for that column. If they’ll’t determine any document in the entire inhabitants, they pronounce that the info themselves are incomplete.
- If a sound document with lacking knowledge is recognized, they then apply the transformation logic manually on that pattern document once more to see why the info for that column didn’t come by way of. Right here there can be 2 eventualities:
- Situation 1: the sampled document incorporates some surprising traits, making their column values excluded from the gold layer. Briefly, this can be a design drawback. On this situation, engineer will talk about these surprising traits with the product proprietor and decide a remedy plan for them. Both they’ll determine that they’ll safely ignore this subset of the inhabitants, as the info with these traits usually are not related for the enterprise goal; Or they’ll provide you with customized transformation logic for them, with a view to convey the info in.
- Situation 2: the column worth comes by way of of their guide transformation, this implies their notion of the preliminary knowledge lineage is fallacious. Briefly, that is an execution drawback. The great engineer will return to test what the info pipeline is doing, or what the info lineage really is. Then they repeat the remainder of the steps.
A nasty engineer will:
- Have a poor understanding of the info lineage.
- Determine a pattern document with lacking knowledge within the gold layer, however has knowledge on the supply for that column. If they’ll’t determine any document, they pronounce that the info themselves are incomplete.
- If a sound document with lacking knowledge is recognized, attempt to apply a guide logic transformation on a document with lacking knowledge to see why the column doesn’t come by way of.
- Provide you with the fallacious conclusions of why the column values don’t come by way of, primarily as a result of their understanding of the info lineage and general knowledge pipeline is fallacious.
- If their commentary leads them to a Situation 1 conclusion as above (a design drawback), they’ll inform the workforce that this can be a knowledge high quality subject and name it a day. They assume the design is ideal and there’s nothing to enhance right here.
- A extra moral but in addition extra disastrous engineer will try and provide you with a customized remedy for the impacted information (i.e. modify the design), nonetheless, they make an even bigger mess as their notion of the info lineage is inaccurate to start with.
- If their commentary leads them to a Situation 2 conclusion (an execution drawback), they’ll return and examine the hole between the carried out and designed knowledge pipeline, and may very well provide you with the appropriate answer subsequent time.
What an evil engineer will do?
- They might or might not know the right knowledge lineage, that is irrelevant.
- They pronounce that as the info for the column is incomplete from the supply (primarily based on what the enterprise advised them), after all, the info can be lacking within the dashboard.
- Then they assume that there isn’t any drawback with the info pipeline, as the info is inherently incomplete.
- They name it a day and go house.
Hopefully, my instance above has given you a clearer depiction of the three varieties of engineers. Nevertheless, the instance can solely help you in the long term after you’ve grasped the basic variations between good, dangerous, and evil engineers. To systematically differentiate among the many three, it’s vital to seek out out their important traits:
Right here is my tackle that:
- A great engineer possesses 3 qualities: distinctive information, dedication to fact, and dedication to outcome.
- A nasty engineer lacks both distinctive information or commitment to outcomes. Nevertheless, they do have a medium degree of dedication to fact.
- An evil engineer has no or little dedication to fact. The result’s of no significance to them. They care about different facets (maybe the looks of outcomes), or they don’t care about something in any respect. It’s uncommon for an evil engineer to have distinctive information, but when they do, it’s not related anyway, as once more, they care neither for the reality, nor the outcome.
A few of you might discover that there’s not a transparent distinction between the dangerous and evil engineers right here. Usually, evil typically does hurt — so you’d count on an evil engineer to introduce malicious code with dangerous intentions, or to cowl their previous errors. I agree with that. But, what I’d like to focus on right here is the place I draw the road between dangerous and evil:
It doesn’t essentially require a malicious motion for the engineer to be evil, as soon as the engineer begins ignoring the reality in entrance of their eyes (i.e. pretending to not see the issues), they cross into the realm of evil.
And the extra info they ignore, the extra evil they’ll grow to be.
So subsequent time, if you meet an engineer, search for indicators of all these three qualities. Don’t be so assured but in the event you simply discover a record of credentials, certifications, or many years of expertise — they’re simply indicators of outstanding information.
Dedication is an energetic frame of mind. To search out indicators of dedication to fact or outcomes requires cautious investigation of historic behavioral patterns, steady evaluation of 1’s considering course of, and observations of their reactions towards challenges.
Neglecting to search for indicators of dedication or fact is to neglect your individual success, and let it’s determined by the supposedly ‘knowledgable’ engineers.
Ultimately, that is about you being accountable for your individual hiring/partnership determination. In case you don’t wish to waste your cash, begin figuring out the nice, dangerous, and evil engineers.
***
Hello there, in case you are studying this, the possibility is you care about knowledge. You suppose there are invaluable values that may be extracted from knowledge, and you’re keen to seek out one of the best methods, implementation practices, and instruments to extract as a lot worth as attainable out of your group’s (or your individual) knowledge property.
If that’s true, then try my weekly e-newsletter — Knowledge & Past Dispatch. Each version will convey you insightful content material from the Knowledge group, which is curated and summarised to provide you contemporary, well-articulated, and sensible views on the mission, visions, methods, and toolboxes of actually efficient Knowledge Leaders.