I was hired 6 months ago as a freelance data engineer, and after proving myself through my work quality, I am now essentially functioning as a tech lead, with full responsibility and ownership of designing, implementing, and hiring for the projects I'm assigned.
Our company is not a tech company, so I only have a couple of tech-oriented colleagues, and I barely interact with them. Now I directly report to the director of the company, who in all senses is awesome, with 40+ years of combined experience in some of the biggest oil and drilling companies globally.
However, I have some strong FOMO about not being able to learn much technical stuff from my peers or seniors. I am trying my best to learn and pick things up on my own, learning design principles, getting code reviews from chatGPT, etc. But even then, I'm afraid I am not producing the software to the highest standards of the industry since we don't have any rigorous cross-checking, and might be missing out on a lot of learning.
Can someone who has been in positions similar to these please guide me?
1. Learn how to learn well, continuously, and sustainably. Tech changes rapidly. And you will want to hop from one domain to another, just for keeping things interesting and to move with markets. This is both a blessing and a curse. It is a blessing because you can start late and still be in the top percentile if you have the brains and work hard for it. It is a curse because you will be doing this no matter how many years of experience you have.
2. Hone your non-technical skills– caution: these are compounding over time (both good and bad habits) – being disciplined, thinking clearly, articulating clearly, being professional, being trustworthy, managing your physical and mental health, being dependable/reliable, having a growth mindset, thriving in ambiguity and uncertainty etc. then, honing your communication skills – effectively collaborating with people, give/receive effective feedback, do/get mentoring/coaching, working with cross-functional people, working with very seniors, very juniors, peers etc. read a lot, develop mental models, deeply craft your personal approach to first principles problem solving, to making tradeoffs/bets etc.
You can do the above all by yourself, through reading, and observing people from afar, and engaging with people (even strangers on forum like this one) in dialog.
In code you don't have all the extra communication avenues that we have when speaking, like body language, intonation, sarcasm and so on.
On the other hand, when writing code we are not speaking in real time. We can think about a problem for a while, consider the best possible way to solve a problem and how to explain it.
What do you see as a big difference?
(3) Communicating clearly is an orthogonal skill to coding clearly. I think this skill is barely acknowledged in engineering cultures in comparison to the above.
I feel you have to have an engineering culture that values institutional knowledge retention, team education and growth — and not treating engineers as fungible — to get to level (2). Level (3) would be a great place to work.
Taking a complex domain and effectively communicating it (correctly) at different levels requires having not just rote knowledge but an actual understanding.
Then in a regular work I explicitly detect where it pays off and feel “see I told you”. This creates a motivational loop to continue not-adhd-ing through tech.
Sometimes I still fly over the knowledge, but then may note that what I’ve been doing in a complex way could be solved with one parameter, if only I knew about it. This creates negative feedback against flying over.
This is ofc only one facet of learning, but I find this “see I told you” method very effective, cause my main issue with learning is unwillingness to learn for no clear reason.
Now, I mostly actually enjoy doing this and thus it has not really limited me. But I wish I could just spend some actual work time on more ‘non adhd-ing’ what I learn.
Sadly I'm now so easily burnout that even setting a dev env up can burn me out.
For example, I discovered early on that I learn in three phases: 1. I get exposed to something (a concept, a process, etc); basically discover that something exists. 2. I then see how that thing is used whether through mentorship or tutorials or, increasingly, through trial and error. 3. I apply that thing to some novel problem.
Through this cycle of Discovery-Tutelage-Application, I can assess my level of comfort with new material and understand when my struggles are due to trying to short circuit the process.
It's likely that you have some form of learning process that is equally cyclical, yet undefined -- once you identify and codify those steps, you can evaluate your progress when it comes to acquiring new skills.
To make that more actionable... My approach in life has generally been to find a project (even something seemingly incredibly dumb, as long as it is fun), then work through it, learning what I need to know as I go along. To learn "well", you must then also constantly question what you have done as you complete various stages of the project to see if you have done them as effectively as possible, and try to incorporate any lessons learned into future projects.
I have found that how individuals do the learning required for this differs significantly from person to person, so it is hard to recommend any particular approach.
I am not sure there's any hand holding that can be given to someone to learn how to learn.
There's keeping the engine well maintained (the sleep and exercise part, for example) and there's driving the engine down new paths and honing you driving technique. You work on the latter by exposing yourself to novel and interesting arguments (interesting philosophy or argumentative non-fiction, for example) and then working through the argument again with counter-arguments in mind. I would not recommend pop-sci books for this, because their arguments and writing tends to be quite flabby.
I'd actually recommend something like RG Collingwood's "The Principles of Art" which is a relatively plain English example of well written philosophy:
https://archive.org/details/in.ernet.dli.2015.188470/page/n1...
You will disagree with him. The point is to understand how and why you disagree with him and to explain that clearly, not only to yourself but others as well. Thinking clearly is about responding and communicating clearly while displaying a sure and succinct understanding of the problem at hand.
You can apply that to everything you read or are confronted with, but the key thing to realize is that "thinking clearly" is something you practice. There is no one trick --- it's an approach.
1) Start contributing heavily to a popular open source project you're familiar with and use. Try to make your PRs as high quality as possible. You'll get free code reviews from some of the best engineers in the world doing this. That'll be a million times better than ChatGPT code reviews and you'll be able to learn a ton from it while also getting your code into production at thousands to millions of companies in the world that depend on the open source project/s you choose to contribute to.
2) Fill holes in your knowledge base. If you feel weak in a particular area like networking or DSA then study and practice with it until you no longer feel weak in that area. If you were on my team I would try to assign tasks to you to help you naturally build out knowledge and confidence in your weak areas but without someone to do that for you, you'll need to try to figure out your own weaknesses.
3) Always try to do your best when working professionally. This is all any of us can ever do and if you actively practice it then it'll become a habit that will guide you toward success in any environment.
Not sure about the other points.
But, sure, don't send PR's for things that don't matter just to send PR's, that's just annoying.
But if you send a 3-line PR to fix a bug, that required 15 hours of study of the code to diagnose and figure out the right 3-lines, that is HUGELY educational.
Working on immature or even poorly written software can be super educational too, nothing wrong with that too.
But if you only ever look at hacky not-yet-mature not-yet-proven software, how would you ever learn to write high-quality software, or even know the difference? You have to study known good code. The way programmers study something with a natural goal instead of just artificially, is in the course of trying to alter it "correctly", the study you must do to figure that out.
(I know some programmers who have literally never worked on anything but over-engineered architecture-switched-every-month-for-years barely-holding-together technical debt bankrupt software... and I think sometimes some of them literally don't know anything else is possible! Looking at mature respected code is crucial, if you don't even know what it looks like how could you possibly try to get there?)
Getting your own work critiqued is unmatched, but reading what others did is the next best thing.
As the other poster pointed out, heavily contribute implies the project still could use some major new features and there is plenty to do. Even porting something KDE to QT6 would be a major effort and teach a lot even if you don't write any new code. There are projects that are complete as well and so there isn't much to do.
I wouldn't say you'd only learn specifics of the frameworks. Maybe if you only submitted 2 or 3 PRs, but that's not what parent is suggesting: 'start contributing heavily'.
It wasn't fun, but I'm better for it.
Read about your technologies. A lot.
When I started working (about 3 to 4 years) , I spent 20-30 minutes a day reading DZone articles about Java, software design, architecture, OOP. Persist and do it daily. Repetition and habit are key.
And be focused. Understand everything in every article, do not accept not understanding everything. For each article, try to find out whether it makes sense, try to understand what the author wants to tell you. Think about how you would have done it. When possible, apply what seems to be useful and overcome the limitations of what is written /proposed in the article.
Occasionally, do the same with IT books instead of articles.
Then explore further, for example, find out how certain popular OOP patterns may be replaced by other, equivalent, patterns in FP. Think about how OOP classes are equivalent or not to closures in FP.
Also, become an expert in fundamental practices, like transaction management.
Application of theory is key.
That's how you do it.
I often see people either get caught up in theory without practice, or proclaim that it's all about experience. Actual time spent writing code is probably the most important thing, but having some foundational ideas and getting to skip some things you've have to painfully learn yourself by reading and then getting to apply them directly is huge.
I'm guessing in practice it's probably comparable to music and language learning, which seem to work similarly in this respect. You do need to spend time practicing by doing, but having that be most of your skill development time is sub-optimal for people who aren't already at an advanced level. Most your time should actually be spent carefully observing what the experts are doing. Because that's what builds the intuition you need to critique your own performance. Which, in turn, acts as an enormous force multiplier for your active practice.
That's arguably why contributing to open source projects is such a great method. Because, unlike drilling yourself on practice problems, the vast majority of time spent contributing to an existing project will be spent reading and understanding the code that's already there.
Excellent code is slow to write, the first 10 times, but eventually it's just as fast as junky code.
You can create a PR that coworker 1 loves because it hides the complexity and then have coworker 2 come in and say that it's complete garbage, because while the complexity is hidden, it still has to be understood in this context, only increasing the difficulty of working with the project
I.e using a state-machine vs a switch/case to determine a status attribute.
Both viewpoints are valid. Some code is objectively terrible, but most discussions on code quality are massively overblown
Or making/sharing LinkedIn posts showing the uber-obscure or complicated "Senior Developer Solution" next to the more straightforward and easy-to-read "Junior Developer Solution."
I’d be careful with that. It’s a great way to get lost down the rabbit hole and works against being focused. Eventually you have to get to a point where you say to yourself that it’s someone else’s problem beyond here and double back.
I read my first programming books that way. Always ask "Why is that word here" and "Why is that ! there". Made me see details.
What works for me is deliberately testing something that I recently read in a project. Even if it is something that works, rewrite it from the perspective of what you just learnt.
This makes you look at problems from different perspectives. And if you get stuck, it's easy to ask for help.
You can learn from everyone around you, regardless of their status. There is no "universal developer experience curve", everyone has more or less knowledge on a field or with a specific tool/framework.
You can learn almost everything alone - I mean learning from the web. There are great forums, groups, discord chats, ask LLMs carefully and check on the answers. It may sound reassuring that someone watches your back and won't allow mistakes or would help clean up a mess, but you should not keep relying on this anyway. Learning by doing and taking responsibility will make you much more self assured, which is actually most of what makes someone senior.
This assessment totally devalues the experience of having an more senior, more capable colleague to learn from. They are not there to watch anyones back, stop mistsakes, or clean up anyones mess. Rather the point is that the junior member is able to watch the senior person, observe how they solve their problems, and how they approach fixing their own mistakes. It's about tacit learning, not bein nannied/micromanaged.
Watching a senior is not a senior pointing you to pitfalls, or unknown unknowns.
The parent argues that although there is value in being near a senior, all the actions lay with the junior. This is a transition people need to make when they leave school and enter work.
You can have bad mentors/seniors, but looking for people to learn from and bounce ideas off is always a good idea.
It might be a personality thing though. I’m a stubborn idiot who questions everything he’s taught. I don’t take advice. I listen to people stories. The why has always more important than the how.
I’ve been fortunate to have many teachers and mentors, but I didn’t seek any of them out and none of them guided me. They were people with the right perspective and I had asked them the right questions.
But the two best mentors in my life are my two best friends. All three of us are different and have different things to teach each other.
But I think there's a difference between someone who 'teaches' you and someone who is at a high enough level to discuss ideas or issues with you.
For example, in my first job I 'learned' the most not from the senior engineers around me but from another new grad who was just as curious as me. We would discuss issues we're having, brainstorm solutions etc. It was not so much that I learned from him, but that I had another brain of similar competence that I could fling ideas at.
Not having such a person makes your job more difficult (as you might find yourself in a rabbit hole and desperately need someone else's perspective) and much less enjoyable.
Imagine not having a competent dev in your team to review your code.
But if you are the only technical person around, who is going to show you what a good or bad practice *in your specific field* is? That you won't find on Stack Overflow or by asking ChatGPT.
Being able to talk to an experienced mentor who knows the field you are working in is invaluable. Unlike learning some framework or design patters or what not, this information you won't find anywhere else.
I've found that one of the harder aspects of being unguided is figuring out the unknown unknowns.
You might stumble into a solution of sorts that mirrors a best practice but not know there's a "name" for that solution -- until you see it spelled out after googling around. That discovery can lead you down a rabbit hole where you gain fuller context.
Sure, having more experienced people around can help expedite that process in some cases, but then again you're limited by what that person has experienced. There's always some level you reach where you need to be curious enough in your explorations to seek out the next layer of knowledge in a self-directed manner, and the tools today are immensely better at supporting that process than 10-15 years ago.
fuzzing
unittest
scm
code coverage
if youre programming without those, youre doing it wrong, and chatGPT isnt going to helpany more im missing?
Fuzzing is only really useful for a very narrow range of analysis scenarios. If people understand threading properly: code should be able to take getting hammered, exiting gracefully, and cleanly get re-instantiated.
Also, banning hosts/accounts with an error-rate quota system is more common these days. =3
this trend in programming culture reduces our ability to do automated error detection!
you make a good point, and a good case for crash early and crash often -- with choice of erlang style recovery, or fuzzing style hard nosed correctness enforcement
is not a fallacy because it doesn't work, it's a fallacy because such teams rarely exist. I have worked on one or maybe two such teams in my career, but the truth is to get on a team like this you already have to be exceptional.
If you require a team of elite experts that are not only technically excellent but also love teaching untrained juniors in their ways, then you it may take you longer than the typical span of such a career to find these people. Then, even if you do find such a team, they're extremely unlikely to take you under there tutelage unless you yourself show an impressive potential to learn. And people that are truly exceptional don't wait for the right teacher to start learning.
I started tutoring my roommate in college, and random people in the computer lab while he was finishing his homework. So I’ve been doing it longer than I’ve been professionally programming which is a long time.
https://onlineteaching.umich.edu/articles/the-myth-of-learni...
There's a good back about how to learn effectively written by cognitive scientists:
Make It Stick: The Science of Successful Learning by Henry L. Roediger III, Mark A. McDaniel, and Peter C. Brown
Definitely worth a read, even reading a summary on the book would be beneficial if you want to become a better learning. The techniques in the book work IME.
Throughout the book you keep hearing stories about how application and spacing are critical for reinforcing concepts.
I don't have it in front of me but I remember they presented one study that students who took a practice exam did better than those that just read the material and took the exam.
In the book they mention often that rereading material isn't useful or an effective way to learn.
Whereas most advice you read on social media is to pay for a course.
Sounds like the ads of our time. I.e. influencers paid to do promotions. Not unlike the ads for certifications and courses before really. Certifications are BS. Someone with zero experience but "certs" is not gonna win any favours with me in an interview. Show you can do the job. A university degree is somewhere in between as there are many ways to get one. The BS - cert kind of - way and the "real" way. It seems the simplest advice would be to just read the documentation then make things, when you struggle refer back to the documentation.
This has always been how I have learned. Learning by doing. Even in university. Just practice basically. Do whatever you need to learn or remember over and over with some time in between. Just like any other muscle exercise: reps!Spaced "repetition is the mother of learning" is what my parents always told me. Minus the "spaced" part.
And yes, that's how we learned for most (good) university courses. We had "labs" every week. It was non-mandatory and while some profs would give you credits towards the final grade if you took part in them, the idea really was to get you to practice things over and over with some time in between. By the time you were learning for the final exam, it was already at least the second or third repetition of applying the knowledge.
Reading docs (or the book / your lecture notes) when trying to apply the knowledge is OK. Just reading it multiple times but not actually trying to use it really doesn't do anything in my experience. I passed my initial "learn a programming language" course at university without ever going to a lecture past the first two for example. I was using said language to actually build something that required what I knew was gonna be the main part of the exam (boolean logic - duh - and concurrency - way more fun) for building some actually useful tool for a friend.
Having a good mentor or two is pretty essential because most of knowledge isn't written down and retrievable by LLMs or about some framework or tool. It is the experience of people who have been there before, done it, got burned and learned to not do the same mistake again.
There's a big difference between learning from someone and having someone teach you something. The latter expedites your progress and clarifies learning path, whereas the former can even waste your time with political fights pulling you into dead-ends.
Realistically, you’re not, but does that matter at all to the company? I’ve had your role in a small company, the job wasn’t really about the tech. Unless there is a dedicated product manager, that is your role now too, and it’s one to take seriously.
You have enviable freedom, do not waste it. Be straight with your boss and explain the tradeoffs regarding availability / tech debt / accuracy you may be making and why.
Above all, understand the business, how your boss sees it, and anticipate their needs.
In this era "High quality" is an increasingly rarer exception. This industry is stuck in the "rush to market' phase even for decade+ projects.
The industry is so young it's still filled with pretenders below-mediocre developers. The fact you're thinking about quality and learning probably puts you on the top 10% of developers in the entire World.
- I'm quite well paid here (4x more than my peers)
- I have been hired full-time, and get all the company benefits as well.
- The work life is great, and I don't work more than 8-9 hours a day.
- The manager/director is great and super supportive.
- I'm not being exploited by any means :)
Kindly upvote so the disclaimer stays on top of the post!
When working in an industrial setting priorities need to shift away from cost optimized garbage people obsess over like chatGPT, and understand when you mess up people may get seriously injured or dead.
Understand shell-offshore standards compliance rules, and explosion rated equipment certification process. Also, integrate the safety procedures in place so when accidents do happen people know the steps to enter safe-mode, and who to phone etc.
There are a lot of unspoken rules, and corruption is commonplace in many places.
To be honest, understand most petroleum related work will be a knock off of Kongsberg and or Rockwell Automation products. There are also dangerous crazies insisting they know something by skimming stolen CVs etc.
Consider the clowns that accidentally cut someones arm off naively touching a hydraulic control-system under maintenance (true story.) Inherent safety is the #1 skill you need to work on, catch the worst case failure modes with sanity checks, and user interaction logging for accountability.
YMMV, and no I don't work for free... Have a great day, =3
People are fragile, maintenance/inspections need done, and team communication is more important than any lock with a post-it note. People mess up, and other people bear the consequences. The "figure it out as we go crowd" is not welcome... =3
You're supposed to remove your lock when you finish up, but it sounds like sometimes angry phone calls have to be made.
Study the old engineers that still have all their body parts, and never feel pressured to do something silly.
Best of luck, =3
1) Read code and spend an hour a week keeping up with developments in your immediate field. Understand the most used frameworks, learn how they work, what each one's pros and cons are, the lingo, the discussions.
2) Understand what your manager cares about and what he wants from you. Ask him what would be a success and what would be a failure.
3) Make sure you understand and implement best practices. Do you have a golden set? Tests? Monitoring - Do you know how much data runs through your pipeline, what your error rate is? What wakes you up when the pipeline fails in the middle of the night? Who needs to be notified, what are the downstream effects?
4) After you have a couple of projects under your belt - that may already be the case if you've been there for a while - consider hiring an experienced contractor for a week to review your code and architecture. You said you're responsible for hiring, I'm guessing you have a budget for subcontracting. Like with all contracts, make sure you have well-defined scope for what you're asking for.
- Learn the domain you are in deeply. Talk to the experts (not senior engineers, I'm talking about people doing the work) in your company about how the business and industry works.
- The interface is the product. Design beautiful interfaces that match people's mental models. A beautiful architecture needs to support these interfaces. If you have poor engineering, it will show through the product via the interface.
- All good engineers are good architects and product people. They understand the needs of their users. They understand their psychology.
- There might be software patterns for your industry, find them and learn them.
TLDR; The coding part is easy, don't seek the senior engineer to get ahead, look for the industry and business experts. Learn how to build products that are impactful for the people using your software and the business. That's how you become a great engineer.
> fresh graduate data engineer
to the company, they see this as "cheap labor"
> hired 6 months ago as a freelance
basically, they put you through the ringer without having to pay out benefits
> essentially functioning as a tech lead
a fresh out of college graduate is far from being qualified as "tech lead", you are out of depth here
> with full responsibility and ownership of designing, implementing, and hiring for the projects I'm assigned
basically you are a 1-bus factor team. hope you have negotiated pay accordingly.
> Our company is not a tech company, so I only have a couple of tech-oriented colleagues
you are a second class citizen at this company, your budget will be constrained, and be on the first to cut when it gets tough. In the O&G industry, the boom and bust cycles are quite frequent.
> Now I directly report to the director of the company
now there's internal pressure to perform at a much higher level than your experience.
If you want real advice, get the hell out of this company while you can. O&G industry itself is notorious for back stabbing/conniving.
Just to clear up some misunderstandings (caused by my own lack of information supply TBH):
- I'm hired full-time with benefits now after working freelance for 6 months - I get paid 4.5x more than my peers in this company - I got these many responsibilities after proving my competence and delivering on reliable solutions in my freelance tenure. - Yep. Once again, I'm a little overpaid. - Also, currently I am being treated quite well, and definitely not like a second class citizen, since the company is making a shift towards a tech-first and data driven approach and I'm at the centre of it.
You have definitely made a few great points here though. I'm out of my depth here when it comes to management, very high level design, etc - but given the scale of operations so far, I am quite sure I can deliver outcomes!
Also, thanks a ton on warnings about the nature of this industry, I'll definitely keep it in mind!
The Oil & Gas industry is not sexy and never will be, but it pays well above market and people with niche skills will do very well there (as you have no doubt now seen).
Technological progress in the space tends to move pretty slowly (especially on the Operational Technology side of the business) owing to many factors, but mainly risk avoidance. The flip-side of this is that once one company adopts a technology, the others tend to follow suite, so what you're doing may be very portable should the need arise.
Another benefit of O&G is that due the the specialised nature of a lot of the roles, you'll keep running into the same people at different organisations throughout your career, so keep your keep expanding your network along the way.
Don't let your perception of slow growth worry you too much - on the technology side, progression will come naturally as you take on new and more challenging projects. As a tech in a very different field, but a similarly isolated environment, I too have had these thoughts throughout my career, but remember there is a big wide world outside of silicon valley, and plenty of successful careers to be had.
It sounds like you have a boss you like who you could learn a lot from - this is huge. I'd focus on getting him to teach you to become a domain expert in the Oil & Gas space. This combined with your (relatively) niche technology skills will make you a unicorn in the future and combined with your network, pave the way should you want to become independent in years to come.
Good luck!
Best comment
Then for your skill, understand that usually data is the most important thing in a system, especially for internal company. Learn about database design and normalization, as a data engineer I'm sure you know how critical it is. As for application level, ensure that ACID is achieved on critical-level operation, and learn maintainability.
Additionally backup / disaster recovery and/or replication also can be learned if you interested on devops route more. Having a backup (and can restore it) is a must, either you or hire someone else to do it, or hosting at a cloud.
your comps are with “working in San Francisco”, not with your peers
There is truth here -- yes, OP is likely out of his depth a bit. Yes, he's likely to be seen as a "cost center" rather than an asset. Yes there's internal pressure to deliver at a level higher than experience.
But none of those are reasons to flee.
OP: I think you would be wiser to enjoy this situation, learn as much as possible, deliver as best you can, BUT be fully aware that you could be nixed at any moment and do NOT take it personally if/when that happens.
Another thing to do, on the other end of the spectrum, is not get "stuck" at this company. You might find yourself extremely successful there! But there is a natural limit to how much you'll be able to learn/grow as a solo operator at a non-tech company.
Use the time you have to learn as best you can and pivot to a role at a proper eng-focused org where you can get the mentorship you want (assuming you want that!). Find another gig before you have time to learn bad habits or get too stuck in your ways.
2 years at this company will likely be great for you and be an asset to your overall career. 10 years will likely not.
Keep fighting the good fight.
I would like to stress that your employer should, and will likely be happy to pay (or give you the time) to do many of the following. Don't feel like you need to do all of these by any means, I just wanted to give you everything that came to mind that has helped me in my journey:
- Conferences both in your specific domain (in your case oil and gas), or your specialty, like a more general data science conference. Meet people there and keep in touch with them, both fellow attendees and also speakers and people who have booths there. Your employer should be cool with covering one or two of these a year. If they're reluctant or on the fence it has helped me in the past to writeup a little summary of the conference, what I hope to learn and how it apply to my work, and give them a written summary when you get back including concrete things that you learned that will help your company pragmatically.
- Meet mentors on platforms like this, foster and build those relationships. It seems like you are already doing this well. Other places to meet them are at local and virtual meetups for the software or packages you use, tools you use, industry societies, etc. Linkedin is also an underrated and underutilized tool for this IMO.
- Free or low cost remote courses. I'm just wrapping up a remote class at Stanford, it's been awesome. Many universities offer courses, or even course materials and videos you can get your hands on. Good universities, with good courses.
Do these things during the work day, because they are work. Don't think like you need to do all this stuff above and beyond. Any good company should be investing in you.
-
1. Turn to online communities to help you learn. Find the appropriate Reddits and Discords, hang out on Stackoverflow, join Bluesky/Mastodon and follow a bunch of people in the same space as you.
2. Start teaching others. The best way to learn is to teach. Answer questions on Stackoverflow that you know the answer to, help people out in Discord.
3. Start a blog! Even if nobody ever reads it, writing about what you're learning is an incredibly accelerant for understanding things more deeply and retaining more knowledge.
4. Conferences. Ask your boss if you can get a budget to attend conferences relevant to your work. See if they'll shell out extra for workshops.
5. Side projects. If you want to learn something new and you don't get the opportunity at work, see if there's a small side project you can spin up to start experimenting with it.
6. Find people in similar roles to you at other local companies and see if you can get into a pattern of meeting up every month or so to talk about what you're learning and share notes. Obviously don't go sharing trade secrets with the competition, but in my experience there is always a huge amount you can learn from talking to peers in other companies, in an environment where people feel safe to let their guard down a little and talk openly about problems they are facing.
Branching out like this is critically important even if your company has senior folks who can give you helpful feedback, since your own company will have its own biases.
Most of which can be distilled into: Don't follow fads, the latest shiny tech; they will only break your heart and maybe ruin your career. You want to deliver a quality product that just works, never breaks down, and if it does it is easily diagnosed.
What that means is (a) use software from vendors you can trust and have actual live support (this means they are usually expensive) (b) KISS. In a small shop you will be responsible for everything. You want as few moving parts as possible. You especially want to avoid all custom integrations; they suck. If software B doesn't offer an out of the box connector to software A which your company already is using. They are immediately ruled out. (c) never trust a salesman/saleswoman. Before you sign any contract, you want to try it out as proof of concept. Just because they say they have an out of the box connector, for example, does it actually sync the fields you want? Verify it works before signing any contract. If the salesman doesn't like it, tell them good bye.
When setting up my current operation, I needed a job scheduling framework. I looked at Airflow, but by the time I had to choose a DB backend and create usernames and set up SMTP server, I gave up. In half a day I knocked together a basic single machine scheduler, which does 5% of what Airflow does. But it does that very well, took me frankly less time than setting up this behemoth, and works really well for my team.
Is it better than Airflow, or Prefect, or whatnot? Surely not; but it saved me time and taught me a lot about job scheduling.
Something that turned out to be extremely valuable was having the chance to make lots of technical decisions (many big, many teeny tiny) and then stick around long enough to evaluate them afterwards.
A lot of things worked out great, and a lot of things didn’t! And now I have a decent amount of insight and opinions on those sorts of things. And those insights getting me a lot of respect at my new job! Which is only my second job, after being in the hot seat at my first, which was a startup.
So my advice is try to at least stay around to evaluate your own decision making. You will end up challenging a lot of your own beliefs. I think it changed me more than anything else at that job.
Sticking with tech sounds like a safe option, but if this opportunity was given to you, maybe you could think about rising in the rankings within the oil industry.
I’m not saying this is the only choice, but this is a choice that you should consider as well.
Also, learning from more senior colleagues is really a hit or miss. I’ve been in environments where I couldn’t learn anything from senior colleagues and where I learned a ton working with relatively junior people.
Despite the technical knowledge is important, the domain where this is applied is even more important. It is always good idea to keep oneself in and field, instead of going from fintech jobs, to retail, to B2B SaaS...
One of my colleages has always been working in the energy market field, and he is way more knowledgeable about everything involving the business than any other technical peer. Also, he has a name in the industry, and is known in most of the energy vendors of the country. It's reputation alone has made more than one contract for my current company, so it really pays off. Of course, he is also paid well above any of us.
The fastest way to learn is to move to the Bay Area and work with people who have been doing this for decades. That is not a sufficient criteria (anyone can be a bad teacher), but the experience is extremely useful.
That sums up anyone's college experience.
The hard part is telling apart what doesn't matter from what does. More often than not, what dictates which is which is the project you find yourself working on.
If technical proficiency was all that important to your employer, you probably wouldn't be in position you're in.
YMMV, though. Some data engineers are writing basic SQL or playing with Azure Data Factory and there isn't too much complexity. Read Designing Data Intensive Applications. If that sort of thing resonates with you then find someone to work with who has experience with those kinds of problems!
[Edit: no disrespect to ADF! Just pointing out that the data engineering discipline is broad and different practitioners will have different expectations of complexity]
This 100%. It seems that you're doing things proficiently enough to keep your job, but just want to improve your capabilities as a coder. It's going to be difficult without a proper structure or mentors around you. So you've to make a decision whether you'd prefer to stick with the role and get slowly better over time using blog posts and AI reviews, or jump in the deep end and get another role in a more tech industry aligned company. And if you decide to jump to another role, just be aware that you're going to spend 3 to 6 months panicking about whether you're in over you head, and probably won't get "comfortable" in the role for a year.
Ideally you should also run those linters in CI/CD. When a new member joins the team, they will get CI/CD linting for free, which will save you a ton time for each new team member
I wouldn't worry about this too much. As long as what you're doing seems to vaguely match whatever information about best practices you can find online, you're probably good. As long as you're not inventing your own security tech, using floating point values for financial data or reusing feature flags in a high frequency trading scenario, any bad decisions made along the way probably won't be catastrophic and are more likely to be learning experiences.
More eyes on code is a good thing, but can go too far where a person is cargo culting their beliefs and subjective experience as objective things, leading you to learn the wrong stuff, or at least to a degree where you have an irrational attachment to some practices (e.g. some people always lean towards EAV in DB design, even though that practice has a time and place, same for microservices vs monoliths vs modular monoliths etc.).
> ...and might be missing out on a lot of learning.
This is probably what you should focus on. See if you can attend industry events, watch some talks done by people who've been in the industry for a long time, there's actually lots of nice stuff on YouTube that's not just advertisement for the latest technology or practice. Here's some people whose content I enjoy:
Dylan Beattie
Venkat Subramaniam
Kevlin Henney
Maybe look at some of the better received books. Here's a starting point: A Philosophy of Software Design
Algorithms
Building Secure & Reliable Systems
Clean Architecture A Craftsman Guide to Software Structure and Design
Clean Code
Code Complete
Design Patterns Elements of Reusable Object-Oriented Software
Next-Level Database Techniques For Developers
Patterns of Enterprise Application Architecture
Peopleware
SQL Antipatterns
The Clean Coder
The Mythical Man Month
The Pragmatic Programmer
Working Effectively With Legacy Code
Again, take none of that as gospel, but maybe give them a passing glance.Also, props to whoever mentioned learning by doing in a low stakes environment (side projects) and online communities!
I think I read about this story
Keep a tech journal.
In it, describe what you're working on, what problems you're trying to solve, and questions you have.
Make conjectures based on what you know and what you guess, and write those in the journal too. Come back later and flesh out or correct those conjectures as you learn more.
As your writing reveals holes in your knowledge, add those topics to a list of things to research.
This kind of journal is much easier to keep in a wiki or note-taking app like Obsidian, since you can then build a personal knowledge base around it. However, you can also do it the old-fashioned way, with pen and paper. Do whatever comes easiest, since that's what will stick.
Do exercises. Take 30-40 mins a day to do practice problems. Use a site like codewars, leetcode, even past advent of code challenges. Do the exercises in the text books you read.
Write. Write about what you learn. Use your writing to explain it to someone who is encountering the subject area for the first time. Doesn't matter what you write it on: a paper notebook is just as serviceable as a blog. The act of reinforcing what you discover by writing it down helps to reinforce it and challenges your knowledge by explaining it to someone else. Whatever you do make sure that the act of writing works for you: it must be seamless and flow easily.
Don't over-engineer the perfect writing system that use a ton of frameworks and require maintaining deployment scripts. You will avoid writing if it is hard to do. Writing itself is hard enough and that's what you need to focus on. If you want to go the blog route make sure it's no harder than opening up a text file and running a simple script when you're done.
Lastly, find a mentor. It can be someone outside of work -- and in your case sounds like it will have to be. Contribute to an open source project you use regularly. Go to meetups in your local area or online. Find someone you can talk shop with. And when the opportunity arises ask if they'd be interested in meeting up once in a while to review your work or talk about what you've discovered.
Ask them what the original goal was, how that changed, what proved to be important, what was unexpected.
BTDT (Companies too cheap to hire experience, and I'm pretty smart). I am also a high school dropout with a GED, so...cheap.
In my case, I remained very humble, voraciously ate up all the literature I could, and got my company to send me to many seminars, conferences, and classes. I did a lot of traveling, and learned to be frugal (lots of cheap motels).
I made a lot of mistakes, too. My company was fairly tolerant of that, because ...cheap, and I also had a lot of successes.
"Good judgement comes from experience. Experience comes from bad judgment."
You have a marvelous opportunity, here. Stay humble, learn like crazy, and don't be afraid to make sure that your managers know where you're at.
Good luck!
Damn that's good
I think Will Rogers and Rita Mae Brown are credited for using it.
You have been given an opportunity to build and lead team, hire the person you would want as a technical mentor. Focus on the big picture results, what will you be able to put on your resume? Built and Led a team that developed a technique/software/product that yielded $x result for company, or “worked on x product in y language”
It’s a different class
If someone who has a successful company wants to interact directly with you, learn everything you can from them, that is an opportunity greater than any tech bullshit
Having an insatiable drive to learn is the only requirement! Keep reading, keep doing.
Would I change it if I could? No. I think I’ve built up a strong will and understanding of the craft. The opinions I have are self formed from doing. I like that.
I at least tried to reverse the situation by doing whatever I could to help mentor my juniors over the years. But it must be said: just because someone is senior it doesn’t mean they’re good. You can easily end up learning dogmatic bullshit that takes years to shake off.
So I always say to juniors “take my words with a pinch of salt. I might be wrong. You need to find out what you believe on your own. Don’t just rely on my words alone”.
In your situation you should have an honest and frank discussion with your boss. Tell them you're feeling a bit out of your depth. Explain that you think you need more senior heads to help guide the project. He/she'll either take it seriously, or not. If not, leave and find somewhere else to work. If you don't have this conversation then any failure will be yours alone. If you do have the conversation, then you've moved the ownership of future issues to your boss.
Also, working in a company where the product is what you write is the place where you'll learn the most.
So in short, see if you can contribute to a well-run open source project with a good community.
Pairing and doing code reviews together with someone, even if they're not more skilled than you, is a nice way to feel better about technical work. Catches bugs well too.
Seeing if you can learn from seniors in or outside the company is worth it. Sounds like you won't get detailed 'here's the right algorithm for X' help but that's not the worthwhile bit of any mentorship IMO - the useful parts are more often 'here's a way to push back on unrealistic requirements / work with colleague X better / handle conflict.'
Finally, remember that there aren't some mythical grownups who Do Things Right(TM) - everyone is making it up as they go along. You're already valuable to the co, and you'll keep getting better with experience - blink and you'll be the senior producing to 'highest standards' and giving all the new hires impostor syndrome ;)
> However, I have some strong FOMO about not being able to learn much technical stuff from my peers or seniors. I am trying my best to learn and pick things up on my own, learning design principles, getting code reviews from ChatGPT, etc. But even then, I'm afraid I am not producing the software to the highest standards of the industry since we don't have any rigorous cross-checking, and might be missing out on a lot of learning.
On having the right mentors:
This sucks, and I think the rest of the comments have good advice (look to open source, read more, etc.). I'll also say, think of the upsides. You have a lot of autonomy. You have a lot of latitude to implement your ideas. You have the ability to ship something and see it fail, learn from that, and improve it.
At some of the big companies, there are so many talented people hungry for good jobs or empire building that it’s really hard to have your ideas implemented.
On high-tech quality:
I spent many years working at Airbnb and Amazon. I have friends at Facebook and Google. The honest truth is that all of these places have lots of bad code. The entire industry is held together by duct tape and glue. If Google (clear tech leader of the 2000s-2010s) and Airbnb (Tech-darling built in a post-AWS environment) both have "bad" code, where is there "good" code?
1) Companies that succeed solve business problems and don't focus on fixing tech debt that doesn’t fundamentally matter to customers.
2) Real businesses are complex and don't fit nicely into good design patterns. Every big refactor and rearchitecture I saw at Airbnb started with good design but got complicated once they hit reality.
3) Good engineers deliver good results despite bad code. Take the time to deeply understand your tech stack. Figure out which parts of the code are brittle and figure out ways to improve them. Make the case to your business partners that fixing things will improve developer time, reduce mistakes, and save them money.
And you’re in your first job, not your last job. Take the time to learn the ropes, and when you feel like you've outgrown your role, find a new one. There’s always something to learn.
Real Q is, do you care about oil and gas? Probably answer is no. This sucks too.
The skills you gain from autonomy might translate into, you have a good idea for a new business, you ask your rich boss for a $25k-$100k check investing in your new thing when you leave. I assume it will not be in oil and gas because nobody really cares about mineral extraction sincerely.
This deal sounds nice but in reality there are millions of people your age who get a $25k-$100k check every year from their parents to do something they care about, as opposed to some boring ass business.
Your instincts to find mentors are good. The hard truth is you will have to quit to find them, you have correctly ascertained that you are hitting a ceiling, and developing a relationship with these guys is kind of worthless.
This is the real value of “Google” and “Airbnb:” they’re so big and they’re household names, even if you worked on some bullshit meaningless thing there, you can get a lot of people to pay attention to the new thing you want to do that you are actually excited about. The skills you develop will not be useful for getting hired at these companies, unfortunately that is a huge crapshoot right now, and you’d have to do a career reset by taking grad school or something, because I cannot emphasize enough how little people care about mineral extraction.
If you aren't experienced already it is a very hard position to be in, with a huge responsibility and potential to screw up due to inexperience - and be promptly thrown under the bus when the proverbial shit hits the fan. Code reviews by ChatGPT can't compensate for lack of more experienced colleagues.
The best way to grow as a fresh grad is to join a medium sized, established business. There you are going to be a part of a team where you will get to learn the ropes of your field (that's something you won't find in ChatGPT or university) and at the same time the pressure won't be so hard. And while you aren't going to get the perks of working at huge companies like Google or Facebook, you likely won't have to deal with assorted corporate BS that comes with it either.
Only once you get a few years under your belt think about startups, small companies, etc.
> The best way to grow as a fresh grad is to join a medium sized, established business. There you are going to be a part of a team where you will get to learn the ropes of your field (that's something you won't find in ChatGPT or university) and at the same time the pressure won't be so hard.
My experience differs: already at medium sized, established companies there is a lot of corporate bullshit. Also, at small companies, you know the colleagues better, so that the culture is often more "humane". Also, exactly because at a small company you are a "jack of all trades", you by necessity learn a lot more; but be aware that on the other hand you won't specialize so deeply into the topic. So, if you want to deeply specialize in a topic, you better look at a medium or large company (but be aware that there might exist few or no jobs there that allow you to specialize in the topic that you are into).
Remember the point of industry software isn’t the software, it’s solving business problems. “Right” technically and “right” for the business are sometimes not the same thing. The only way you can learn to tell the difference is by keeping a good pulse on the rest of the company, and looking at your past decisions in the light of whether or not they enabled its success.
If you can't, find some courses, and participate in open-source development (where they need help, and will often provide feedback). If you need rigour, try test driven development. Attend meetups and conferences and try to meet people there.
I've spent my commute time to learn about these topics from youtube. For example watching EEVBlog or Uncle Bob or whatever. Then stop sometimes and just think about it regarding my past experience. Replay the stuff that I lived through while thinking about what I heard. Then find my best colleagues and sincerely ask them what they think about what I was concluding just then.
Also I like Twitter/X after just muting/notinterested-buttoning the people who just post idiot stuff. Then it is quite a good source of ideas about the present mainstream - that's also very important.
But the most important: never forget while swimming in ideas and memes: take some minutes sometime to asses what actually worked - and what is that somehow never works for ... reasons.
Great question on HN!
It sounds like you have the initiative to push your own boundaries, and that's what matters most. No matter where you are, learning starts with your willingness to learn.
I'll admit I've never been in your position precisely. I've always worked with technical people, even if not for "tech companies." But, before I had even graduated I was the only person in my entire company doing mobile dev, so I know what it feels like to be very green, and very far out on your own island. It was a great springboard. I never did mobile dev full-time again, but I experimented and explored stuff I found interesting, and the skills I learned have been invaluable to my career.
I don't know what kind of languages/environments you use, but I recommend investigating and trying out some popular technologies and architectures that are allowed in the scope of your work. Try to understand the pros and cons of each approach.
When you want to know the "right way" to do something, try reading the code of the most experienced programmers and successful projects written in the same languages/environments as yours. Very little in software is "new" so chances are someone has tried it before. Maybe LLMs are good for this, but I usually just read the code.
Focus on iterative improvement not destinations, because you'll never be "done" you just move onto the next project and the next technology. You'll get better as long as you're trying to learn new things.
Mentoring - As you build your network, you'll naturally find opportunities for informal mentorship. This could be someone with a solid career who has already navigated the tough paths and can help you tackle challenges while also offering guidance for your journey.
There have been many times in my career when I felt stuck. By having regular conversations, listening to others, and sharing work experiences, I gained a broader perspective and have always been able to found ways to move forward.
I am self taught and had no mentors for the first decade of my career. During that time I did things nobody else could do, things I was told were impossible.
Eventually I read some books and they added useful tools.
Read all the books from SICP to Superintelligence (ie concrete to speculative fiction).
But nothing is as educational as putting what you read into production, or putting it into production and fucking it up. I’d rather work with someone who has deleted a production database (ideally a preproduction database to be fair) than someone who hasn’t.
1. Learn algorithms, buy some book and start reading.
2. Learn theory of neural networks, use the Chollet book to develop some knowledge and intuition.
3. Write many small "pet" programs implementing basic stuff: implement a small database, a small programming language interpreter, a small editor, a small neural network, ... Each time try to apply good design.
4. Embrace simplicity in everything you write, don't make things more complex than needed.
5. Read good quality code, especially read open source code which is at a degree of complexity that makes reading valuable. Dont' read Kubernets... or PostgreSQL perhaps. Learn more self-contained code bases.
6. Participate to some OSS poject.
7. Start some side-project and put it on GitHub, were you are the main designer. Develop something that you need. A library, or a small utility, something that you really enojoy doing. Do it at a quality level that makes you happy. Never regret pushing code on GitHub if you feel that for the level you are right now it is appropriate.
8. Don't give a fuck about what other programmers think of your work, if you did it at the max level of what you can do. Anyway most programs on the Internet suck, including the ones of people feeling very competent.
There's lots of good advice in this thread, but I think the most important thing to keep in mind is to know that you will make mistakes and do stupid stuff. Always be evaluating yourself and use that to adjust how you do the next thing.
Anecdote: in my first real job after school, I was essentially give a one person R&D project that reported directly to the CTO to design a framework for "the future of our data analytics". I ended up designing this crazy over-engineered highly abstracted OO framework that was a nightmare to document and use. But since it was fast and met the requirements, the CTO pushed it through as the framework for the rest of the engineering team.
I spent the next couple of years justifying it and supporting it. Once I recognized that I made many dumb (well maybe not dumb, but novice) decisions. I used that as a learning experience for my next project.
I've had many of those kinds of learning experiences over the years, and as long as I keep self-examining and being self-critical, I learn from them and don't make the same mistake twice. Now 25-ish years later, I have a long list of internalized knowledge of what not to do.
I guess that's a long way of saying, don't be afraid to make mistakes, you will. Just pay attention to them and learn from them.
Failing that, I’m not sure what your relationship with your manager is like, but in most tech companies, part of your manager’s job is to help you grow. Maybe they could help you reach out to the few other tech people at the company to organize some kind of collaboration to help you all grow and improve.
I am interested in the other answers here and will add my two cents.
Recent hiring: KEEP IT STUPID-SIMPLE. Being new at the company, less experienced, and empowered, you will probably be tempted by lots of "solutions," whether they're products, frameworks, management buzzwords, etc. Most of them are not designed for a one-man-team. Make sure your solutions actually help YOU.
Non-technical boss: AGREE ON METRICS. Your boss/es will measure you eventually, whether either of you are aware of it or not. It may be better to discuss some metrics while you have a chance to do something about it. At the end of the day, it's almost always about money (and sometimes ego), especially with cantankerous O&G types. Just figure out how you can show you enable, save, or make money within the system.
Feedback and standards: YOUR BEST FEEDBACK AND STANDARDS WILL COME FROM YOU. You are in a position to act on your best judgement, and then own your mistakes. Yes, you can feel like a fish-out-of-water a lot of the time. I have tried to find a balance between pushing myself to learn personal skills on projects, innovating for the company, just getting things "done," and keeping myself sane. Your peers, "clients", and supervisors are probably not used to (or aware of) "best practices", which is a double-edged sword. They might expect you to deliver things you know could be better, but you also have a ton of power to build a solid environment for yourself. Think of progress like driving a big boat, not a racecar.
Join Discords related to the stuff you’re working on, you’ll find people much smarter than you and any of your colleagues hanging out, talking about good approaches to designs, structures, infrastructure, etc.
You’ll also find idiots not worth listening to.
Remember you’re 6 months in / just doing the job is enough of a challenge. You’re calling yourself a lead for having full responsibility of the projects you’re working on. This is typically common everywhere except large tech companies.
Read and study and learn from your mistakes and the mistakes of others – you will believe things and realize later that your beliefs were wrong.
If there was one book I had to recommend that I have read it would be Sandi Metz's 99 Bottles series.
Oh, and learn multiple programming languages.
There's no way not to learn from this, and there's things you can learn here that you wouldn't learn from years of being in a regular team. So just as you're missing out on valuable experiences here, you'll also be missing out on valuable experience if you change jobs.
The interesting part of a career is also all the non-technical people we meet, and if you're learning from them, that's also good. Maybe this road will take you somewhere more interesting than spending a lot of time on code reviews.
(PS: If this is actually in oil, make sure you're getting paid well.)
Self-directed can absolutely help. (Though, a recommendation: do a lot more reading of human-written articles and a lot less chatting with AIs.)
But you also need to find some other people to talk to, both peers and people more senior than you along one axis or another. You'll learn from them and they'll learn from you. Open Source projects, communities of practice, professional organizations, friends in other companies, online communities, etc.
As the most senior technical person in the company, you can treat this as an opportunity to create paths for technical and process development for your technical (and non-technical) colleagues. If you find opportunities, make those opportunities available to your colleagues as well, or pass on what you learn in one form or another.
My tip is to try and best practice everything. Locate the correct method online, especially if theres applicable vendor best practice. Every time the director asked me to do something new, I would go away and find out what I should be doing, and then made that the basis of our negotiations. I didn't win every one of these fights, but I managed to spin the job so far that I did IP telephony, Security Cameras, built the company's ad forest and hundreds of other small relevant to my career wins. I ended up with 2 direct reports. And in the process I took a very immature business and injected a large amount of maturity that they kept with them for more than a decade. And I did them correctly, or as correctly as I could without support. By the time I looked to move on I was immediately made a senior in my next role.
Here's what I recommend:
1. Build a corpus of materials: Conferences (in person, and you can youtube them), books, courses. Cross product these concepts with tools/techs - DBs, coding tools like IDE and source control, programming languages, programming paradigms, software craftsmanship, network stack / protocols, debuggers, Observability and Ops, Peopleware / wetware. 2. Think of attending conferences as seed materials for future research, write down every term, subject, and technology you're curious about, never heard of, are scared of, think is out of your league/level. 3. Spend an hour a day on weekdays, and 2-4 on a sixth day, sharpening the saw[2] (learning) 4. Close loops on subjects by taking action. Eg: If you finish a chapter on a certain DB, install it, insert some data, learn to bulk insert, try some complex queries. If it's a language try to write a breakable toy[3]. If it's a paradigm, try a code kata and write the same one in multiple paradigms. 5. Remember: you read a text book a month in Uni, try to finish a text book (or equivalent effort) per quarter/half for the remainder of your career.
[1] - These are definitionally top .1 or even 0.01% engineers (amongst the engineering population, not the general population)
[2] - Sharpen the saw - https://www.franklincovey.com/the-7-habits/habit-7/
[3] - https://ilango.hashnode.dev/learning-by-building-breakable-t...
I have a few years of experience under my belt, in a small team of ~10 devs. No one is a software engineer, and it's everyone else's first job essentially. There's no one senior from whom to learn, and I feel stagnant.
I improve by interacting with the open-source community. I have been getting more and more involved in several projects over the years, even making it as a maintainer for a few of them.
I recommend you look at smaller communities -- my favourite is Haskell -- as compared to very established ones like Python's. Smaller communities mean that you have more impact as a contributor, which requires you to keep pushing and learning new things. Smaller communities make it also easier to interact with senior people, for example on Reddit or a Discourse instance (e.g. https://discourse.haskell.org)
I suggest this high level idea, which I have done in isolation in the past and caused tremendous growth. Think of something you'd like to optimize, run an experiment over a year and take careful notes. For example, once I decided I wanted to minimize defects that I released into production. For a year, I tracked every single defect I released into production, and made brief notes about how I could have avoided it. This led to me completely changing my attitudes and approaches about what is important. I discovered, for example, that very few of my production failures were caused by me writing code with bugs, and that by far what I needed to do is improve my testing strategies, what kind of data I test with, and how faithfully I represent the production situation during unit testing and end to end testing, and how I approach testing in general. There were a lot of observations and approach changes I made, but the thing that made these really stick is that they were my own lessons. Everything in my process was rooted in a hard lesson I learned myself. Whenever someone asks me why I do X I have a war story to back it up.
The point is, you can figure out for yourself the big areas to improve if you set a goal for a thing you want to improve, and carefully record your results, and review those periodically and look for commonalities. You can look at the advice from others endlessly, but the best teacher is experience, usually your own. Make the most of your own experience by figuring out what's important to you and think about process improvement for yourself and those that you work with, as a first class thing.
- Learn other frameworks and/or programming languages. It gives you a wider and unique perspective on other "correct" ways of doing software. Through finding commonalities one can arrive at things that are good practices regardless of the tool being used. Through finding differences, one can better understand what might be dogmatic to a specific language/library/etc.
- Dig deep. If something isn't working, as much as possible, make it your goal not to fix it, but to fix it and understand why it was broken in the first place. Trying things until something works is a tried and tested way of fixing things, but digging through the abstraction layers and confirming why that suggestion one found online is very valuable. I think a lot of knowledge from senior developers boils down to having experience at lower levels of abstraction.
As someone who was a developer and ended up in a support role by mistake of company
Now i left that company but here’s how I relearned everything
- Read lots of Code, look how things are implemented. You can use LLMs just don’t blindly use them to copy paste ask the LLM to thoroughly explain the approach.
- Treat LLMs as someone who has lots of Knowledge whom you can ask lots of questions.
- looking at different open source projects is really helpful and having the hungry mindset of learning
- How will the company / your boss decide that you are doing a good job? - Focus on that first. Include the parts they don't directly value but where you think you need them in order to reach that goals.
- How do YOU know you are doing a good job? - to develop that taste you need peers to look up to. Find them in meetups, in un-conferences, conferences or on the internet.
- Build your skills in three main areas:
- "Behavioural" skills (a.k.a "soft" skills) - written communication, verbal communication, be good at giving and
receiving feedback, building professional relationships, organize your work, learn to "manage up" - domain skills (learn about the company & the field)
- tech skills - focus more on concepts & principles and less on the "framework of the day".
Not impossible but much harder than working closely with humble and experienced engineers. That's why I recommend that a junior engineer finds a great consulting company as soon as they can and spend a few years there. This was a kinda breakthrough period in my career for both technical and soft skills. The nature of consulting also means that you might see a greater variety of environments, technologies, and problem/solution combinations than the equivalent xp in a non-consulting role. But make no mistake, finding a "good" consulting company is key. You need to have humble and experienced engineers who want to help you and grow you. If it's a "bums in seats" kinda "body shop" or some kind of celebrity house for "hero" or "10x devs" then maybe this won't be as useful.
2. Do that by reading, here and other useful places, not necessarily about your own area.
3. Start to understand your industry, its details and quirks, watch your directory, ask them questions, learn the business in detail. Even if you move to delivering IT in another industry, the knowledge will hold you in good stead.
4. Understand how to model business workflows without worrying about technology. Learn how to communicate that with your users and customers (two different groups sometimes).
5. Nouns are more important than Verbs. Concentrate on the "things" and how they react to events and generate their own.
6. Technology comes and goes (MVC, DAO, microservices, monoliths, k8s, etc etc), processes as well (6 sigma, CASE, Agile, UML, etc etc). Learn their important details and differences and don't think any of them are silver bullets.
1 & 2 are the most important.
What I did:
* Read. All the things. Read HN comments, read tech blogs, read books. Don't put too much weight on any single book/comment/blog post—they'll almost all disagree with each other, and you'll learn the most by comparing and contrasting all of the strong opinions rather than uncritically absorbing any given opinion.
* Be okay with using work time for professional development. Your employer hired someone they knew wasn't experienced into a role that is typically filled by someone more senior. They need to be okay with you learning on the job, and if they're not you need to go somewhere else.
What I wish I'd done but failed to do:
* Don't get overly invested and burn out. This is your first job, but it's not your life, and your career doesn't depend on you completely blowing it out of the water as a brand new developer thrown into the deep end. Don't overinvest.
* Keep your eye out for red flags. Tiny companies that can't hire seniors might be small and niche but sustainable, but they might also stay small because the business side of things just doesn't work for various reasons. In my own case, it turned out the founders of the tiny company were both narcissists (which seems to be a common problem with tiny companies), which led to a ridiculous political implosion that I was lucky to escape.
My thought process was I get to learn stuff, so even if I didn't make any money off it (I mostly didn't), I learned how to ship. This included (yuck) documentation, QA, etc. Doing things like that you really learn not only how to code and how to structure your programs, but about the 80/20 rule which to me is it takes 20% to get POC and 80% to get it ready to ship.
The senior I finally got to work with was mainly to validate that I was on the right path, because I could write just as well as he could. I met this guy probably 4 years into my career and we were pretty equal at that time. He had about 5+ more years experience than me, and he was VERY sharp. I probably haven't worked with someone better since and this was 20+ years ago.
One more thing, I was the "computer guy," at my first company out of college. I did the workstation building and support, the networking, the server setup and user administration (Windows NT days), and wrote applications in MS Access for the company to use. I had to interview the users and managers on what the apps should do and whatnot. I had to learn how to scale MS Access for 250+ user base (ugh). I had to do all the reports and everything computer related. That company was a toxic shithole but it was actually a blessing. I never would have learned as much had I got a big corporate job. You tend to get pigeonholed in those places doing the same thing over and over. And yes, I was terrified of not being good enough the whole time; that helped too.
Good luck!
Practical tips on learn good techniques, do research, find the best tech companies that do similar development to yours. Check out their technical blogs, their githubs, find opensource projects which have been developed to the highest standard. Dig into them and potentially even rewrite your own simple versions to learn, maybe twin it so you could make the new implementation a part of an internal research project... so main possibilities there.
You need some example, some role models to look up to.
And it's usually not about some tech details (you can google it), but about some important style of work, ways of thinking, approaches to engineering, attitudes etc.
Something that you can't find in books, yet it is very real and important for hi quality engineering work.
At the same time it is not necessary for a relatively successfull career. Most of the companies are fine with mediocre software products (you can see this around you lol) and would not pay more for higher quality work.
So the ultimate answer depends on your definition of "grow as an engineer" :)
I work in a team with some very senior colleagues (and even famous in computer circles for a couple of them) in a reputable company. I actually don't learn much from them. They review my code, we discuss some design docs, it's certainly interesting to see how they manage their time and how they deal with various problems, but I don't learn much from them in the technical sense. Also to my surprise, we also don't produce software to the highest standards...
If you want to improve technically, you can read books on things you'd like to improve, or find technical projects outside of work.
The disadvantage is that it’s hard to grow beyond what you can visualize, so past a certain point you’ll likely be growing slower than you would working under someone with 10-20 more years of experience.
I would advise one of two paths: either stick around a year or two u til you can confidently own “tech lead” on your resume and move on, or commit to a longer stint and really reap the learning benefits of the technical leadership position.
So much for the unsolicited career advice :). To answer the direct question, a few things I found useful as a startup CTO:
- Follow the news for your tech stack. Stay up to date with the various technologies. Go to meetups. - Think about the long-term development of your system. Spend 1% of your time thinking about where you might be in a year or two, and do some experiments to explore those paths. Your codebase is the best place for you to experiment with new tools as it’s a real-world system with the warts that reality brings. You don’t have to merge these experiments! As CTO I had probably 20% of my PRs as RFC / experimental work. (I do not advise contributing to FOSS for learning in your scenario. Feel free to do so if you enjoy it though.) - keep an Architecture Decision Log. Review the big decisions you made in hindsight, and learn the lessons that you can only learn if you own a system for multiple years. Eg this DB choice seemed good and bought us a year of scale reprieve, but then required a 2 man-year rewrite down the road. Knowing the true value of “I” will make you much better at judging tradeoffs in ROI. - Beware burnout. If you are studying and leisure coding in your spare time you will eventually run the tank dry. It might take 5 years or it might take less. Just be mindful. - Don’t be terrified of burnout. The tech lead position is a unique opportunity to feel ownership of a system and have the autonomy to try things out. It’s a role where pushing harder can yield more leverage than a normal position. So putting in some extra hours can pay off in the long run for your career.
The way I excelled from beginner to a higher level engineer was self learning, curiosity, building things so I could understand them. Example: when learning dependency injection, I found a tutorial on how to build your own DI container. So i built one and saw how it worked internally. That was SUPER useful.
I recommend Podcasts, YouTube Videos, Courses, Conferences, Participating in Open Source (even writing docs that are missing is helpful). Ultimately its about using what you learn and always being curious, learning patterns, and what is useful vs what is not.
The last thing you want to do is learn from a senior. Any worthwhile mentor that actually is worth listening to will usually tell you to go try stuff and figure it out.
If you have a laptop, you have all the information and environment to learn. The only thing that you need to have is a framework of learning, which is very simply this:
Chose a feature to work on -> Write some code -> something doesn't work -> research the error, figure out why it doesn't work -> try various solutions until it works.
Rinse and repeat.
2. Get a textbook for a subject which you have taught yourself. Read the table of contents. Assess which topics you know cold, and which ones you have blind spots in. You can assess your mastery of a topic this way.
3. I learned the most by making bad choices and then having to live with the consequences. I learned the second most by having to justify the corners I had cut to other adults. Neither one is fun, but both teach you to be better.
However, if you learn well by doing, or by reading, there are loads of other great ways to improve technically. I’ve made big leaps forward in my skills by building (relatively large) side projects, where I can safely experiment with different design decisions and see the consequences over time. I’ve also got a huge amount out of just sitting down and reading the docs for tech I’m interested in - some frameworks (like React) have fantastic resources that can take you from good to great.
Good luck!
Because you might feel unsure of yourself, you will be tempted to hire folks who are less junior than you - resist that. Hire senior people (budget allowing) and start your interviews with them by asking if they are comfortable reporting and working with someone with less experience than them. As long as you are humble and want to grow, you will.
I think the recommendation to hire seniors and learn from them is great, but I would not muddy up waters by asking the above.
Most senior engineers have reported or saw colleagues report to non-technical leadership. Senior is practically synonymous with being able to make technical decisions without technical guidance (only business guidance). Asking that as a minor question in the middle of the interview is OK. Making that sound too important may just raise worries that the interviewing company had some people issues and might be a mess to step into. My 2c.
If you’re the smartest person in the room, yea, that might be a red flag.
But, here’s some different advice though: get the company to hire someone more experienced. Be willing to give up some oversight responsibilities to that person. (Or hire someone who is experienced and explicitly doesn’t want to play the politics game.)
I felt like I was in a similar situation fresh out of school. It wasn’t until my company hired an incredibly talented person that I felt I was finally learning from someone (beyond reading books… back in that day, they were still pretty relevant and up to date).
Why?
I once encountered a tech lead who was at the same job, since graduating, for about a decade. All of the novice mistakes showed. Because there was no one there to correct basic security mistakes, the lead grew very accustomed to getting their way and getting constant flattery in the process.
When the time is right, seek a position where you're generally the most junior employee, and seek to learn from everyone around you. It's important that you can learn from others before you get into a leadership role.
Standards can slow you down, which is fine in a big tech company, but probably not good for building value for your company.
It's better to think more about what may break in your code base that could be emberassing for your company and build integration / othere tests for those important cases.
Otherwise generally simple code is good code, keep building and have fun in life. Lot of those ,,standards'' just overcomplicate things and may hurt your company.
I find that it is refreshing to hear the perspectives of working and aspiring programmers, and I stand to learn a lot about the field outside my little bubble within it: what various startups are doing, the problems they're trying to solve, etc.
In short, if you cannot find humans to talk with and bounce ideas off of at work, do so outside of work (but be careful about your company's trade secrets, etc.). Do not rely on ChatGPT. ChatGPT is to discussion what ultraprocessed food is to food.
You can use a lot of modern Internet resources.
I got a Learning O'Reilly Account. There you can find courses, videos and live events where you can ask questions. It's helpful.
Read a lot of good books/e-books, ask in forums and maybe in your town is a user group for that.
Just write the software and constantly look to improve. Your code from 2 years ago is in an ideal situation, much worse than your code now.
Find some software that helps your company, and don’t be afraid to dive into the codebase. There may be times you can even contribute as part of your job.
Open source projects maintainers very likely have their own day jobs that are unrelated or only marginally related to the projects, have different priorities or just don't care enough. A previously important feature that has seen lots of activity may be on the shelf, but that's only known among maintainers with no public notes. You don't get to go into meetings or just send a Teams/slack message to get a quick response, let alone a casual chat at the coffee machine.
Unless you are working on prioritized features on a project like linux, VSCode or Chromium, chances are that your issues or pull requests go in the log until someone works on it months or even years later.
Speaking from real experience.
Prepare for the day when workload increases to the point where you are hiring in your first resource. You are now the senior in the equation.
This can be an exciting time. What’s the first thing you’ll hand over? Does it make sense the way it works now? What would you do differently?
Ask yourself these questions and make the changes now, while you have time. Prepare for the team you don’t have. Yet.
It's probably not polite to ask, but I would try and form and answer the question of why that's happening - while it can be a learning experience you dont want to be the smartest person in the room (ever) when you are just starting your career.
I was in a situation where I was able to spend on avg. 14 hours a day for ~3 years programming without a single day off, trying to up-skill myself as much as I possibly could. Fortunately I already had much of the context and foreknowledge of what I needed to be learning, so all that time was spent building; instead of trying to learn what to build, or how.
After some time, I was given an opportunity to join a tech startup, and excelled. I wrote the vast majority of their code bases, I manage virtually all of their ops/infrastructure/deployments. I mentor our junior developers, I conduct all our technical interviews, and I make our implementation/design decisions, even tho we do employ other senior developers with many more years of experience than I have.
Many would be skeptical of someone with so little experience on paper being in such a position, and it honestly I believe it will be a very uphill battle you will have to fight when you are given such a high position, when it's time to move companies. To be quite honest I didn't even want the title I was given officially, because I didn't feel it was appropriate from an industry standard. However there is no doubt that it applies if you read the relevant job description.
As to your question specifically, seek out and spend time with more experienced engineers whenever possible. I am very lucky to have friends with many, many years of industry experience working for large tech companies, who I have been lucky enough to be able to work on side projects with, as well as having been able to contribute to open source, as well as building my own projects and taking on contract jobs on the side where I have solo developed a fairly substantial project as well. I am very grateful for these friends and I have learned very much from being able to build with them.
Build as much as you possibly can, work on projects that are far outside the scope of what you do for work, and take on challenges you otherwise wouldn't feel ready for. Try to befriend experienced devs and seek to be a part of communities where experienced engineers are found. There is no substitute for the hours of writing code, but surrounding yourself with others who's observations and code you will be able to learn and benefit from is a good start.
My first thought is... why? Have you tried reaching out to them to discuss what you're building? Maybe a shared chat channel (slack, teams, whatever) where you can discuss what you're doing; or even rubber duck and people can read as they want and offer their insights. Just because you're not on the same projects doesn't mean you can't talk about your work with each other.
You'll get to work with others who are more senior than you and see how they do things. It's definitely helped me a lot.
“Software standards” come and go, changing relatively quickly, so don’t sweat it too much. You will learn from mistakes, and that’s ok (for me, dunno for your employer :P)
This kind of prompting changes the conversation entirely.
Do you want to write code, and get better at writing code? Then probably find a new job with better mentorship.
Do you want to grow quickly in this company and are okay with being a manager? Then stay on your current path, hire some solid software developers to your team, and embrace the responsibility you’ve been trusted with.
It wasn't until I started going to conferences, Lisp/Python/Perl etc, that I actually met people with the same level of brain damage or worse than me.
Both paths lead to the same place, having access to mentors gets you there faster I guess.
Perhaps they feel equally isolated.
- Conferences, not my thing, but if you are new, you may learn a trick or two there. No just go there, try talking to the people. Approach a few senior looking guys and ask them for advice.
- Confs can be quite expensive, a cheaper alternative is local user groups. You can try to find the closest ones via Meetup.
- ChatGPT (yup, hallucinates but still has a reasonable answer for 90% of the questions).
As for your situation:
- You are responsible for hiring. Try to hire people with more experience than you.
- "to the highest standards of the industry" isn't that perfectionism? Most production code has much much lower quality than what we see in open source.
- I think for the time being (the job market is really bad right now), just concentrate on delivering stuff. Learn if you have some time left, but delivering solutions should be your prio #1 in my opinion.
But aren't needed to grow (but make growing so much easier and as such allow you to grow more over the same time frame).
The most important part are:
- self reflect on you work, importantly do take a emotional step back before doing so to (hopefully) have a objectifish self reflection
- if you do mistakes try to understand _why exactly_ things went wrong, instead of just finding a solution or a target of to put blame on. E.g. if some system lead to some issues don't just put if of because "that system is bad designed" but try to understand which aspects of the design are the problem and dig deeper to find out e.g. if there is a reason it's designed that way.
- in general try to learn the underlying principle independent of the fancy branding and terminology it's backed in. E.g. people all the time reinvent and rediscover long term known concepts and them give them new names and say it's something completely different even if it's not. (But also new impl. of old concepts can sometimes make the difference between unusual and a grate solution). E.g. rail track programming (e.g. in swift) is just a reinvention using Monades for error handling.
- there a many seniors sharing their knowledge freely on the internet, the tricky part her is that not everyone pretending to be a expert is one and not everyone who is a expert in one area is one in another. So never blindly follow their advice.
- when you get advice, especially from less trustable sources like the web, understand _why_/_how_ this advice is going to help instead of just following it. Also question the advice, try to find out when the advice might be a bad idea even if it seems to always be a good idea.
- reading documentation instead of just googling (or LLM generating) solutions can help with understanding (through not always, a lot of doc is terrible bad). Now speed wise you might not have time for that but double checking found solutions by checking them against the documentation is something you often should do anyways.
- be aware that even if you feel you are a senior there is a lot of thing you don't know, more importantly a lot of thing you don't know you don't know.
- don't just reflect on work you recently finished go back to older work
- if it's your thing, keep notes in a well organized way and go back to them to reflect how you understanding might have changed
there is more but I have to go
Where in the world are you? That determines your options.
It is so valuable to take the ideas and questions in your head and get them out. You can then have Claude interrogate and problem solve with you.
By default, I think Claude is very conservative and sometimes comes up with ideas that aren't quite right. But you can actually just tell it to be less concerned about X or Y, to start over from scratch, etc.
Another trick I use is to ask it to ask me one question at a time to help clarify my thinking. I find that the one by one questioning really helps me find the boundaries of my knowledge and crystallize my opinions.
Build things. Solve problems on your own, or if you are already in management instruct others to do so. You learn by failing until you don’t.
0. Writing a lot of bad code.
1. Reading a lot of good code.
Fresh grad, 6 months into their first job and working as a "tech lead". Are you leading anyone else but yourself?
I know I'll get down voted but if you were really looking to grow you'd
a) quit and get that junior job more suited to your skill level
b) hire someone to replace yourself and demote yourself to junior role in your own team. (This would not even necessarily affect your comp.)
But seriously though if you're just looking for maximal comp, stay where you are. I'm sure you can become senior director in the next 6 mo.
OP is profoundly lucky. He's working in a hard-to-break-into niche in our industry with super close access to someone that will dramatically level up his niche-industry knowledge.
This positions OP for long-term success. I'm late in my career and can tell you that engineering skill only really matters in a few places and that what and who you know, as well as how you treat people/circumstances, is 99% of the job.
I worked in a different niche earlier in my career and had access to industry leaders and two of my coworkers from that job started their own agency out of it. One of the biggest mistakes of my life not going in with them and chasing tech skill at a different job instead.
OP becoming a problem-solver in his industry will create vastly more career-potential than being pidgeon-holed as a data engineer.
Correct, if they're just looking to catapult themselves to the upper echelons of "leadership" and "executive" careers. But that has nothing to do with engineering which is what they asked for.
engineering
noun
the art or science of making practical application of the knowledge of pure sciences, as physics or chemistry, as in the construction of engines, bridges, buildings, mines, ships, and chemical plants.
Digital Technology. the art or process of designing and programming computer systems:
computer engineering;
software engineering.
https://www.dictionary.com/browse/engineeringEngineering is about making tradeoffs and then you're entirely in the realm of communication. You can pretend otherwise but you'll be wrong and all of the decisions will be made without or around you.
1) actively seek honest feedback from the director of the company, especially in the non-technical areas. It certainly feels good to receive compliments, but it takes courage to ask for weak spots and areas of improvement. The latter will actually help you improve.
2) identify your technical gaps and actively seek mentoring from qualified individuals. You will be amazed to learn that many high performers are more than willing to voluntary mentor ambitious young professionals. Note that usually its very important in those cases to demonstrate the willingness to learn and to go the extra mile in improving yourself.
3) think about what career progression would look like within your current employer. Given your current situation, its important to think out-of-the-box, as a defined career progression does not seem to apply there.
[edit: line breaks added for readability]
Here's my guidance from 30 years in the industry. Stop doing this.
This is a bit of a curse, not the worst role to land on but I would not get too comfy and try get into a tech company in a year or two.
You probably do want to learn some of the basics and force yourself to adhere to them until you are experienced enough to know when to break the rules. Definitely use version control, look up typical styling/formatting/etc conventions for the languages and tools you're using, probably write unit or at least integration tests, prefer to use libraries when possible for non-trivial problems that aren't core to what you're working on. If you want exposure to this I recommend joining a quality open source project's developer discord/slack/whatever just to see how they do things. You could also maybe convince your boss to pay for consultants or for you to attend to conferences or whatever if you want individual feedback (if I were your boss, I'd see this as a good compromise - you get some level of senior supervision for $2k/yr instead of $200k/yr).
My number one advice is to NOT rely on LLMs for things that they're not smart enough for, and to always make sure you actually understand what it's doing any time it creates code for you. This is the number one issue I see inexperienced engineers making these days. ChatGPT is great for converting simple to moderately complex requirements into simple to moderately complex code under a certain size, but it's NOT GOOD at giving reliable high level advice that is "non-obvious" and unlikely to be in its training data. So please do not rely on it too much to explain complicated things to you (unless that thing is very well documented on the internet, which you would have to verify by searching for it anyway) and don't ask to eg tell you if your unit test is good. It can certainly save you a lot of time and is often right, but it's also often wrong in ways that are very difficult to spot unless you know a lot about the subject, and eventually (if not already) you'll need to advance to the point where it can't help you with your problems.
Find other people working on similar/related topics, connect with them, offline or online. Do not fear being THE Tech Guy in the company.. just do not be over-confident / arrogant. Doubt quick-wins, check, prove things. 30 years ago such one-programmer-per-company was a normal thing.
My reading recommendations are below, but that's just what i have found good.. not exhaustive list. i found some of those only after 11 years doing stuff by myself, and saw i have been inventing hot-water in many "wrong"-but-somehow-working ways, so what.. Better than only applying miracle recipes you don't understand.
https://www.svilendobrev.com/rabota/
https://www.svilendobrev.com/1/
And, most important, have fun!
That being said, the one thing that you can't really replicate is getting honest feedback about what you're doing at work.
For example, you can learn all the design patterns, write the best code etc, but are you even working on the right problem? What are the assumptions that you've made that are incorrect? Is there a much simpler approach that'll get you there in one tenth of the time?
These are thing things that experience may provide which no course or single book is going to give you.
You basically need senior people to talk to.
The most straightforward way to do this is to get a job which has this structure set up for you already. Otherwise make friends, go to meetups, do some networking and find some experienced people you can get some feedback from.
Let go of FOMO, youth is the only reason you think it is possible to not miss out. You have and will miss most things because they were not hit your way. The only things you can actually miss are the opportunities that are yours.
But the only way to miss them is because your attention was somewhere else. You have an awesome opportunity for a mentor in the director. You caught a lucky bus, stay on it.
Right now people can look at you and see potential. In just a few years, they will look at you and see squandered potential. Sure technically you are an adult, but you have almost zero adult experience (your mentor has forty years of adult experience). Good luck.
That’s it.
Read books to understand fundamentals. Create projects to practice and expand on the fundamentals. Watch videos to learn how others solved problems. Document your learning. Repeat.
You don't need to FOMO that, really. It works like that in theory, in practice... In about 2 years of work in a midsize company, I learn about one or two little details from the tech leads in the company. Two third of them were spending more time asking why one would think there are worth asking them a question than replying the question itself. The level of rockstar insufferableness was off the charts.
I have a feeling that there is no-one to learn from eventually as everyone is in continuous learning and the industry is fast evolving. In general, there are circumstances where this is definitely not true, but living in it and obsserving it to my current age of considered senior I am still a complete beginner in most of the tech scene tells me this way. My encounters and direct experiences generate a strong bias yet keeping an eye out here and else draws me to this conclusion and tells me that this must be mostly true.
From an other point of view there are lots of seniors. Anyone with 3+ years experience can be considered senior in this industry, given the pace techniques and approaches change (calling improvement would be part lie to a good portion). Which again, tells the same story: not much to rely upon but yourself and keeping a sceptic curiosity about the practices, especially the famed trends. These come and go. Those famed 20 years ago (hehe, more like 10, sometimes 5) are reformed and evolved a lot, mostly past recognition, or most of the time just died. Past techniques considered baaaad practice now, new ones dragging the industry to burning discussion topics and hiring sprees are infants.
The human and social aspects are downplayed in the industry with overwhelming technical dominance.This is where seniority may be a good role model, in mentality and attitudes, but it has no focus. Or sometimes work against it by promoting procedures of replacability: expected work methods and procedures promote the non-dependence on key persons, for the investment heavy industry it is better if the workforce is easy to pick up and lay off. Replacebale parts basically. That is not really an encouragemment for investing in continuity but to show off in trendy areas.
Probably you are already doing your best for doing your best and not making stupid mistakes. Perhaps you could harden your soul like everyone else and not taking seriously after making a stpid mitake, because the future you and all will look back at it and say, this was stupid. Like for so many past things we say now.
> to the highest standard of the industry
I believe I leave this uncommeted despite my deep intrinsic urge, not to attract even more downvotes when I share my mind in this topic, about the 'highest standard of the industry', both as participant and user. ; )
Read books. Watch conference talks. Attend conferences. Find local meetups.
One benefit of this for you is that you have to take full responsibility for your own advancement. As you find things about your work that interest your or that are complicated problems, get excited about the opportunity to solve them because that’s where seniors really come from; deep experience solving and understanding problems.
Good luck!
I was similar to you out of school—ended up the lone techie in a small oilfield services company.
How do you grow as an engineer? Go get a job at a company that does that. It won't be hard. Based on the problem-solving experience you're getting now, you'll impress hiring managers much more than similar-aged / similar-comped candidates, because all they have done is focused on code, and your mindframe is probably somewhat permanently bent toward larger-picture thinking. This is an asset.
It might be the wrong question, though. The question is: are you sure you want to be an engineer? You have a seat at the table in a small business and a mentor who is a strong and experienced operator in the space. Don't under-value that setup for making a career. If you measure yourself against the yardstick of being a good technologist, then yeah, this is not the best setup to grow. But if tech is a skill, not an identity, then you have other goals: building a strong career, producing value in the market and capturing some for yourself as wealth. Lean into playing that game. Ponder how irrelevant it is that some 26-year-old vegan in Silicon Valley would sneer at your code quality. Make it your goal to have an ownership stake in your company or a similar one in the space—maybe one you start or acquire with leverage from your mentor—by the time you're 35. If you get this right, you will earn dramatically more money in your career than most engineers. You will face much better odds of business success than most startup founders. And frankly, the small business operator space is starved for high-end young talent. (It's not comparatively lucrative for the first 5-10 years, and people find the frequent nepotism annoying.) There are a ton of boomers with profitable businesses and no good succession plans. The infrastructure they built—or at least their customer bases—has to go somewhere.
My story: I left the small business I started in and went to a tech company, but I did it because my first firm was terribly unhealthy. (It folded soon after.) Never lost the vision for building and owning something, though.
Learn to write tests and apply a code linter. You will have joined the élite top 10% for doing so. Good luck!
In particular: data engineers' main task is to produce clean data. Few people care about the cleanliness of the code used to produce it. They care about whether or not the data makes sense, whether or not it's accurate, how well it's documented, and whether or not others can understand the ETL pipelines' dependencies and assumptions so that people can predict how changes to business processes and IT systems will affect their business intelligence and data science efforts.
Cleanliness of code is, at best, a secondary concern. Especially if you're mostly working solo. The code for managing data pipelines is often relatively small and tightly scoped, to the point where even a messy ETL implementation can still be easier to understand than very clean application source code. Or at least, that's been my experience as someone who's worked as both a software engineer and a data engineer.
The biggest thing is, again, just to make sure the documentation is good. In one data engineering role I found a bunch of things in the data engineering pipeline that I was pretty sure were junk that we could get rid of to simplify the whole thing. Doing so would have reduced the code for the pipeline in question by ~25%, with comparable reductions in run time and server load. But I couldn't be sure because the original author left no documentation to indicate why those bits were there, so we ultimately decided to leave it there for the sake of safety. The original author was a software engineer, not a data engineer, so the code in question was incredibly clean and readable. I had no trouble figuring out how it was doing what it was doing. But what I really needed to know was what it was doing and why.
I'd also encourage caution about adopting fancy tech stack. It's amazing what you can accomplish with just a SQL database (or Parquet files) and some Python scripts. Data engineering is a space with a lot lot lot of vendors and open source projects selling solutions to problems you might not even have. And every additional solution you bring on is another piece of technology to understand, which can quickly get overwhelming if you're on a small team. Especially be careful about committing to the open source versions of technologies like Apache Spark. I can tell you from experience that, if you can't name the specific person who will personally be responsible for supporting and operating the technology, that person will be you, you will end up spending a lot of time doing it, and it will probably not be a use of time that the person who does your performance reviews will consider valuable. Especially if "I had to spend a bunch of time diagnosing Spark executor failures" becomes the reason why you couldn't get other things done.