*Personal opinions and stories that shaped those opinions are in audio.
Beneath the Bullshit, is The Real BS
When an industry booms, all sorts of weird stuff happens. But I’m here to talk about a different kind of BS. I just have to acknowledge the bullshit part. The real BS is about pain and suffering that one must go through to make AI.
Stories in these articles are somewhat personal and specific to me. But types of scenarios are not. It’s helpful to generalize different types of situations that could occur while building AI and explain how to handle them.
Personal Story is For Credibility
I Started In The Middle Of The First AI Boom
I started in the AI field in 2018 as an intern. This is around the time that I consider to be the first golden age of modern AI. There were 2 catalysts - i) between 2015 and 2016 deep learning frameworks like Tensorflow and PyTorch came out, making it easier to build AI in any environment; and ii) in 2016, AlphaGo beat Lee Sedol, who was then one of the top ranking Go players in the world.
Important thing is that: For the first time, anyone who understands the basics of deep learning could build AI and the entire world saw that AI can definitively beat a human expert. What ended up happening was that until 2020, companies rushed to set up their own AI teams. They did have data, but the problem was that no one really knew what they were doing. And by this I mean that there was no successful attempts to create AI-based products in a way that made financial sense. There were no books, no articles, no anything. A lot of people failed and those who survived have not been easy to find1.
“Thinking Like AI” Gave Me an Edge
“So How Could We Possibly Fail?”
My team had ton of data so we all figured things cannot be that bad. To boost our confidence even more, much of the early team members came out of MIT. Company was even a member of MIT’s CSAIL Alliance, so we could always make use of knowledge and talent. With that much brain power in the room, what could possibly go wrong?
Real BS = Blood & Sweat
Buddhists2 say there are 6 realms filled with different kinds of pain and suffering. People are meant to live in each realm at a time and once they’ve learned everything they could, they move on to the next realm. But if they lose focus, they may get stuck. In my case (and those of others from what I can deduce from observations), there were 6 main challenges that came one after another:
No one in the company, minus a small number of visionaries and the team, knew what AI people are supposed to do.
There were a lot of time wasters - grifty vendors, potential bad faith actors, and just spiteful people - who aren’t always easy to spot.
There was not enough data infrastructure to support all our needs.
Meaningful knowledge sharing with teams from other companies was basically impossible because of lack of incentives or lack of knowledge within the industry.
The combination of above issues really had a grip on the morale of the team: Bad faith actors wear you out, tech debt kills all ego, while lack of knowledge sharing makes you feel lonely.
Most importantly, budget and time allocation were always running out.
You Can Sweat All Day But Feel Like You’re Not Getting Anywhere
Put yourselves into my shoes. It’s your first day and all you want is to just make cool AI models in an industry that no one really thought of using AI before.
Things are always rosy in the early days. But after 3 months on the team, your perception starts to change, drastically.
When you ask why people are sending weird requests, the answer you get from your teammates is that people either don’t know what AI is or in some rare moments they may be so afraid of AI that they might be acting in bad faith. But bad faith actors cannot be proven most of the times.
Rather than being paranoid, you figure out that simple yet the best course of action is to just get good at telling who’s a time waster and who’s not while also getting good at communicating your purpose. This can be tough if your team is full of nerds who have never really spoken to a business person before. This is a slow process and you don’t quite know when you get better at it. Just have to keep sweating.
When You Have An Open Wound, Figure Out How Hurt You Are.
While the team is focusing on figuring out a proper pitch for what the team does, you realize that your current data infrastructure cannot handle large amount of data.
Data pipelines for AI have not been built before and the company doesn’t want to support more resources. What do you do? Someone must build a better data infra but you alone have to figure out how much time and resource it takes.
I can say that this may have been one of the biggest mistakes that many enterprise AI teams made.
Be-Aware Of Vampires And Leeches
Vampires may look alluring but they don’t actually solve your problems but waste your time and money. Leeches just figure out a way to latch on to you.
Easy thing to do is to just ignore all vendors. But there were some serious bottlenecks in the development process of AI (and there will continue to be bottlenecks, just different ones), which may be handled by vendors or at least the ideas can come from vendors.
The best course of action here is to just not get rattled and figure out a way to sort through vendors quickly. Who can actually solve your problem and who’s just pretending? Who has real insights? Who amongst those people with insights, is working on a worthwhile problem that cannot be solved by someone on the team?
This is a bit of a double-sided issue. If one of these people asked us what we’ve wanted, we would have answered better Hadoop or Tensorflow with less errors. And most of these vendors were developing something that looked like the two. But when customers wanted “a faster horse”, Henry Ford delivered a car. How do we know if someone can actually make something you’ve never seen before?
There’s actually an easy answer here, which is just to see if that person who’s trying to solve your problems has actually gone through the same problems themselves. If they can’t think of anything else other than that, that’s a great sign they can really deliver a “car”, not just a “faster horse”.
Be-aware of Internal Bleeding
Getting AI teams from other companies to share knowledge but realizing they too don’t know anymore than you is discouraging. Realizing that companies wouldn’t commit resources to facilitate dialogues feels even worse. Sometimes it creates bottled up anger in the industry. When it’s well managed, it’s just something you laugh about. But when it goes bad, it can destroy morale.
With the environment being really tough to keep up good morale, some people will leave. Working under the assumption that some people will leave within a year is very useful when planning ahead and mapping out what to do - what kind of people can do what tasks, which ones takes training, what backgrounds are good at what tasks, how many different types of tasks do you have, and which will you keep facing 6 months/1year/5years from now.
2025 Is The Right Time To Share
Next big steps in AI are likely to come from individuals who want to solve specific problems using AI. And many of them are going to be working under serious time and money constraints, perhaps even more than what Enterprise AI people have gone through.
Luckily in 2025, we are primed for the next big breakthrough in AI - 1) We have everyone - enterprise and consumers - actively discussing AI in the public, 2) Ability to fine-tune models and deploy to any application just requires someone to program, and 3) No one doubts the value of AI.
A lot of the problems that I faced is solved somewhere. Someone has gone through something similar and can help out. Gathering and structuring knowledge for operational best practices in building AI is the crucial next step I want to fill in.
Future Is Where Children Can Make Their Own AI For A Science Fair
I’m still young, but I feel responsible for the future generation so they don’t have to suffer the same pains that my team and I faced. It’s guaranteed that they will have to worry about different types of pains. So why not let them focus on their pains and not the same ones I’ve felt? Otherwise, there is no innovation and we will be stuck re-inventing the wheel over and over again.
I certainly didn’t go through the pain of inventing a keyboard to be able to write. I definitely didn’t have to go through the pain of developing my own AI frameworks because they were already made for someone like me to use.
What I Can Promise Readers
The soul of this series is that information is open source. Any sort of additional value-add will be providing new experiences with the same information. There are many ways to make contents more engaging.
So here’s what I can promise.
All information in this series will contain 0% BullShit, 100% real-life knowledge earned from Blood and Sweat.
I try to make the contents as engaging as possible within limits of my time - which is maybe 3-4 hours per article: There will continue to be a mix of technicalities, personal stories, visuals, sounds, and more.
It’s going to be informative but never difficult to read.
If I ever ask for money, it’ll not be for the information but the experience. Nothing I hate more than information gating.
What I Ask Readers
If you liked any of the articles, please subscribe. If I keep seeing there are more readers than subscribers, I have to subscription gate all my articles. Knowing who reads what is important so I can make better contents.
Please leave comments and feedback if you have any. Just message me.
Thanks and please subscribe if you enjoyed.
Why? Well I don’t know exactly what all the other successful teams did. But I suspect they never really bothered telling people or because they considered it proprietary knowledge.
It’s just an analogy, I’m not saying anything about my religious beliefs here.
That's a beautiful metaphor about the six realms of reincarnation.