The Change Resilient Podcast Episode 7: Dan Van Tran on the Practicalities of Running Resilient Software Systems
Episode #7 gets deep into building, transforming, and growing software-centric businesses. Dan Van Tran is the CTO of Collectors, a technology company that helps people authenticate, grade, vault, and sell collectibles.
Dan is a technology executive, engineering leader, and CTO advisor who has spent his career building and evolving complex software organizations and production systems.
At Collectors, Dan has led engineering through a period of significant growth and modernization, balancing the realities of legacy technology with the need to move quickly, improve reliability, and build for the future. He also advises CTOs and technology leaders on architecture, organizational design, engineering effectiveness, and navigating large-scale change.
I’ve had the good fortune to work with Dan twice as a customer and I’ve always appreciated his combination of deep technical judgment, pragmatism, curiosity, and willingness to get into the deep details of how software systems—and the teams behind them—actually work.
This is an episode packed full of pragmatic wisdom for the realities of building and operating software systems here in late 2026. Yet, even if you're not a technologist, you'll enjoy Dan's stories of his family's migration from Vietnam, his approach to leadership, and how he finds resilience in the face of AI-fueled change.
Listen on Spotify Apple, Amazon, YouTube, and wherever you enjoy your podcasts.
Links to references include:
Show transcript (slightly edited for clarity, machine transcribed, apologies for errors):
[00:00:03] Steve Semelsberger
Hello, everybody! My guest today is Dan Van Tran, CTO of Collectors. Dan is a technology executive, engineering leader, and CTO advisor who has spent his career building and evolving complex software organizations and production systems. At Collectors, Dan has led engineering through a period of significant growth and modernization. Balancing the realities of legacy technology with the need to move quickly, improve reliability, and build for the future.
He also advises CTOs and technology leaders on architecture, organizational design, engineering effectiveness, and things like navigating large-scale change. I've personally had the good fortune to work with Dan twice as a customer. There may even be a video webinar of Dan and I a few years ago talking about stuff. I've always appreciated Dan's combination of technical judgment, his pragmatism, curiosity, and his willingness to get into the deep details of how software systems and the teams and people behind them actually work. Dan, welcome to the Change Resilient Podcast.
[00:01:07] Dan Van Tran
Awesome, thanks for having me, Steve.
[00:01:09] Steve Semelsberger
So, you know, we've had a chance to work together a couple of times, as I mentioned, and, you know, I've appreciated how you seem to genuinely enjoy the messiness, the reality of complex systems. So, if I got that right, that you like digging in and rolling up your sleeves and, you know, going right into the details, like, where did that curiosity and approach come from?
[00:01:33] Dan Van Tran
Yeah, it's … I don't know … when it started developing, but I did notice it, you know, as I went through my life. I've always been fascinated with how things are built and how they work. And so, just as I navigate through, you know, daily things, I'm just like, oh, I wonder how this ATM works. I wonder how that point-of-sale system works. I wonder how whatever works. And so I basically … reverse engineer an episode of how it's made in my head for all these random things, and probably come up with a way that's not actually how it's done, but is a plausible way of how it could happen.
And so doing that, actually, it has helped me to exercise my muscles for how to solve problems creatively by starting with an end state, and then trying to figure out the different paths to get there. And, I just, have always loved doing that. And, the thing about I got into software, and the thing I really liked about software is that anything you can think of, you can likely create, and there are a lot of different ways that you can do that. And, that's always fascinated me. And also, when you think about it, and we don't often think about this just because the technology's so far.
Computer hardware. It all distills down to electrical signals of zeros and ones. That is amazing, of what we've been able to build with just those electrical signals. And I'd never be able to reverse engineer that, but it's just really awesome to think about.
[00:03:04] Steve Semelsberger
We live in a universe of vibration and energy, and we've created these incredible things off of fundamental physics. It is really, really special. And you know, Dan, through my life, I've met a lot of curious people. But sometimes curiosity gets thwarted easily. So, to accomplish what you've accomplished as a human, you know, as a family member, as a contributor to your communities, as an engineering leader, you've got to have some grit, some fortitude, like, to keep going. Curiosity often isn't enough, so … how do you find the energy and even the personal resilience to stay on a task?
[00:03:47] Dan Van Tran
Yeah, it's, it's something I think about. I get a lot of my work ethic from my parents, and so they, they fled Vietnam during the war, and they went through all this hardship. And then they, came here to build, to allow me and my sisters, who were born after they got here, to have a better life. And so I … it's just amazing that you think you hear a story like that, like, oh, that must have happened, like, a century ago, whatever. It was my parents that went through that.
And so I figure, you know, like, I can probably keep up with a little bit, like, take a little bit of a hit here and there, but … the thing that actually, helps me to keep going is I don't … again, I don't know when this started happening, but I always wanted to leave an impact on the world in a positive way, and leave it better off than how I found it, and then for every area that I … that I'm in, that I'm working on, I want to do the same.
And so I always think about the outcome that I want to see, and then I just keep trying to get to that outcome through various ways, and … When I fall behind in a different … in a certain way, or it's not creating the impact I want, I pivot. But it just motivates me to help to bring a better life for a lot of people to, to, as a technologist, bring new technologies into the world and solve problems in a different way that people may not have solved them in the past, provide better value. But so, a lot of that drives me. And also just the tinkering.
Like, all these new technologies that come out, like AI and all these LLMs, it's, it's just so fascinating what you can do with it nowadays, and so I just try to imagine, what could the world look like, if we were all you know, born into this world with all these tools, how far can we take these things? And so that curiosity just always gets me to continue to try to build things, and, you know, you fail, you just pick yourself up, take the lessons, move on to the next thing.
And so I think a lot of that desire to reach those outcomes allows me to ignore some of the hardships that I experience along the way.
[00:05:59] Steve Semelsberger
What I heard through that were a number of really important things. I heard gratitude, I heard perspective. You know, I heard a deep sense of purpose, you know, I heard humility in there as well, too. Because I think, you know, sometimes resilience can be thought of as just, like, forcing stuff, and it's just like, you know, building the energy to push through, but it's often, as I understand it, a combination of factors that can come together for a human to have enough to keep going, even as things are tough.
And imagine you brought that in as you walked into Collectors, if we shift a little bit into, you know, the company that you're a part of, and you know, I see this as, like, a business that has decades of history, but it's also got a lot of go-forward ambition. This is not like a tired old business just limping along here. So, maybe first you can tell our listeners, who may not be familiar with Collectors, what Collectors does, and even a little bit about, you know, the history and the current moment for the business, and your use of technology.
A lot in that question, Dan, but I think you can handle it.
[00:07:08] Dan Van Tran
Yeah, I think I got this. Okay, so … Collectors. Collectors is a company that was founded in 1986, and initially, they focused on coins, and I believe stamps and diamond authentication, something along those lines. They quickly moved on from stamps and diamond authentication, and just focused on coins. In 1991, they brought in the trading card business. And they went public sometime after that. But the whole business was authenticating and grading the condition of these collectibles.
So people would submit things to these experts, and the experts would give their assessment, and then encapsulate the item with The certificate of authenticity and the condition grade, so that the future buyers, sellers would have trust in, you know, what's being transacted on. So, this was a good business. It was solid, but it wasn't by any means a large, blossoming, booming business. But it did well. And, everything was fine up until right before the pandemic. People started getting into collectibles, and especially because of the pandemic.
Where the sports networks shut down, and a lot of people had all this disposable income, but were stuck inside. So many people started pouring money into collectibles. So, it created this implosion for the company, where they had not built the infrastructure or the technology to be able to handle such a load of volume.
And the one story I always refer to that is quite amazing is, right before the acquisition, they had so many submissions that the local post office actually shut down and required them to rent school buses and funnel all the packages back to the headquarters so that they could continue processing with a regular mail...
[00:08:51] Steve Semelsberger
Wow.
[00:08:51] Dan Van Tran
...coming in. So that created an opportunity, for, Nat Turner, our CEO, who was at the time CEO of Flatiron, Steve Cohen, and a bunch of others to come in, and … acquire the company, take it from public to private, and then start to think about how do we help this company to scale to match the demand? So … you know, over the years, the thing that they did well was build up a lot of expertise and a lot of systems and processes on how to deliver that value consistently. And so that part we, you know, when we came in, we definitely did not want to mess with that.
But the part that we did want to try to figure out is a lot of us came from startups or scaling businesses and evolving them, and so we wanted to figure out how do we help this company to scale. At the same time, there were a lot of innovations that were missed over the years, because they were really just focused on the service, rather than continuing to add value in other ways on top of it. So, we actually started looking, not just on how to, how to improve and scale the current business. And how do we also push it further in parallel?
So … for me, the way that I started was, the tech team was very small, 22 people or so, 11 software engineers, but there were over 300 web domains, dozens of products, applications, no No product team, no data team, no security team. And …
[00:10:29] Steve Semelsberger
How long ago is this Dan, approximately?
[00:10:31] Dan Van Tran
This was 2021, so that's the year of the acquisition. And that was also the year that I joined. So, you know, the company had evolved from 1986 to 2021, using very few technical resources, and actually using those resources more like a corp IT help desk, more than an actual software development organization. So, they had, like, a CIO, and just, like, a very classic organizational structure. Not really anything remotely similar to big tech.
And so when we came in, a lot of us came from big tech, and we saw this, and … wanted to understand how we could Change it without actually affecting the core business, like the actual authentication and trust business. So … the, where I started was the team, in any of these sorts of situations, the teams actually know more, than you do about, where things are fragile, where things are old, where things are actually, okay.
They may not know how to fix it, they may not even be able to articulate when you ask them where are things fragile or breaking, but when you ask them to go through and list out the architecture and draw a diagram, it's clear where they see, like, the potential weak points being. So …
[00:11:51] Steve Semelsberger
...risks, problems...
[00:11:53] Dan Van Tran
Right, exactly, because these are the areas that have caused them pain over the past, and maybe they just put a band-aid on, moved on. But, you know, they know. They know that these are the areas that we need to come back and fix at some point, so … I started with a listening tour, going through to all the different teams, and especially given that, at that point, there were 35 years of infrastructure that was built across multiple business lines.
So I just wanted to understand how everything fit together, what had been built, and what decisions were made, if any of those were documented anywhere. And, the unfortunate thing was that there weren't very many artifacts. A lot of the people who built the initial infrastructure were gone. And there were just millions of lines of code, a lot of it written in VB.net, not even using modern front-end technologies. They were using WPF, for those of you who are a bit more old school on the Windows stack. Windows Presentation Foundation, which is a very old, you know, set of technologies.
And just … there were a lot of other things that were just, they were likely … Decisions that made sense at the time, but were never revisited. So, you know, one of the things that … took as I was listening to that is. A lot of people, they think old, they think that that means something that needs to be replaced. But old doesn't necessarily mean bad, which I think some people link together, you think about buildings. A lot of buildings are old, but it doesn't mean that they all need to be torn down and replaced, right? And so I think that the approach that I have.
[00:13:30] Steve Semelsberger
By the way, to your point, Dan, I'm a child of the Midwest and the Rust Belt, upstate New York, Milwaukee, Syracuse, father from Detroit, son that just finished college in St. Louis, and I've been spending more time the last couple of years trying to go deep into the rebirth of once-great older American cities. I was in Detroit two weekends ago, and there is some really, really good stuff happening in these cities, in many cases, like with Michigan Central, honoring once great architecture.
So keep going on this, this tension that you sometimes have around coming in and seeing things as broken and possibly fragile, but also realizing, you know, where your priorities are. Because I think you're, you know, you're into this tale of, you're building the team, because you've been growing your team, right? You started with a few folks, and how many engineers, or how many people in the technology organization do you have today?
[00:14:27] Dan Van Tran
Yeah, now we're around 240, 250 in tech, about another 40 in product and design. And, we actually have 50, 60 open headcount now, probably 70 total by the end of the year, and so we're growing quite rapidly.
[00:14:42] Steve Semelsberger
I mean, in 5 years, it's like 10x growth in team. So, Dan, with vibrant growth in your organization, like, 10x growth in team, a lot of technology change, the market is changing, how do you figure out how to prioritize things? How do you make small bets? How do you make bigger bets? You know, how do you think about experiments and quick pathways to learning? So, maybe you can, you know, take us on the inside of the Dan Van Tran playbook.
You know, managing growth and transformation simultaneously, because sometimes companies separate those, and it feels like you're in the middle of both of those, and have been now for a number of years.
[00:15:23] Dan Van Tran
Yeah, it's funny that you mention that. I think, I've only worked at 3 major organizations in my career, but each one was a different phase of what you mentioned. So my first company was really just focused on building the future, and just solely focused on that, and that was … not very fruitful. We end up spending a long period of time trying to migrate, and Just build, just import things and, We did not build too many innovations during that time. And so, things were a little bit stagnant.
It was okay, because it was a big, big company, and, you know, the customers weren't going anywhere, but at the same time, it wasn't expanding the company's potential at all. And then I went to a startup that actually had an acquisition of an older company's technology, and we started building on it, but we never really went back to fix any of the old things. We just kept piling new things on top of it, which created different issues. And so that was a challenge. So … when I came to Collectors, and I was in the driver's seat. I try to marry those two ideas together.
And I was thinking, so … how do I do this? Where do I start? And the thing that I had learned, especially at my time at Flatiron, the startup, was that It's always helpful to imagine the future, and what it looks like, the ideal state that you're trying to get to. And then you can work your way backwards to see what needs to change in order to get there, and then you can prioritize those steps. And it becomes a much clearer picture than trying to organize things by product, technology, or anything else. Like, that's just a little too rigid.
Being open with just, here's the ideal state, here's our current state. And in order to get to the ideal state, here are the things that need to happen. You then have a bit more flexibility in what you can do and where you target things. And for me, there were 3 major categories. People, processes, and tools. I always start with people. A strong team will figure out what to do and adapt as needed.
And for me, I just have one perspective, one set of skills and expertise, so I try to find the smartest people in the various other areas that, especially the ones where I'm weakened, and surround myself with them, and then I give them an objective, and then let them do their thing. And so, that was very helpful, and I was very lucky in this situation to actually have inherited a lot of smart people, which I know in a lot of M&A is not the case, but I did have a great team to start with.
[00:18:04] Dan Van Tran
And so they did help to rebuild the foundations. So from there, with the people, I, brought everyone together and just started sharing my vision. Like, okay, this is where I think the future looks like, and … asked them to give feedback and make any changes, and so they iterated on it a bit. And once we had a vision that we all agreed to. I … they all had their own areas, and I asked them to figure out, alright, how do we build the roadmaps for you to get to where we need to go, in your area? And then I gave them support and autonomy.
And a lot of them never worked in this way before, and so at the beginning, it was a little daunting, but then I, you know, for each one of them, just went and … Try to help them to see the steps they could take and the creativity they should exercise, and all of them ended up getting to the point where they were able to come up with a picture of some sort of milestones to get to this new state.
[00:19:03] Steve Semelsberger
And then, the vision, was it a technical vision, a product vision, a customer-oriented vision? How did you calibrate the right level of vision to galvanize the team?
[00:19:16] Dan Van Tran
That's a great question, so … I … This is actually something that's interesting, because I think as technologists and people working in tech, a lot of times. We follow a very strict process in that even our career ladder is already defined before we join. You know, like, it goes from, like, software engineer to senior software engineer to staff software, all the way up, right, to CTO. And so, that is already defined for you, you just follow the ladder. And same thing with technology. A lot of times, you know, you are hired to build something, but you're told what to build, when to build it.
You don't need to think about those things, you just build the thing. You exercise your skills there. But the problem often isn't building the technology. That actually is a relatively easy problem in most cases. The thing that's difficult is figuring out what to build, when to build it, and if it's even the … if it even needs to be built. So, what I've realized over my career is that In technology, it's especially important to start with what does the business do? Like, how do we provide value for our customers, and clients?
And then take the product hat and think about, okay, so what products and services do we need to do to add that value and deliver that value? And then you can start thinking about technology. Okay, so now that we've thought about the products and services, how do we build them? What do we build?
And so, when you think through that lens, I … when I went into collectors, even though some of those things weren't defined for me, I tried to build it as much as I could, the business goals and then the product goals, based on the conversations I was having with the various executives and leaders and our customers. And so then, for me, I was tasked, of course, with the tech goals, and the tech vision. So I sat down to create a vision and mission statement. The goal is pretty clear, you know, we provide these services for authentication and condition grading, but really, the product is trust.
Like, how do people trust our judgment that we're applying to these things? And the products that we had were the services and sites and all that that people can use. So then, I thought about, alright, so what's the most efficient way to deliver these services? what I came up with was, actually. You know, a lot of tech organizations are centralized, you have a QA team, a security team, software engineers, and software engineers are generally broken down by business unit, but everyone else is kind of shared. And that does create a lot of prioritization bottlenecks and issues.
[00:21:55] Dan Van Tran
So, what I realized is, if we were to move efficiently, these businesses are so different. Coins versus cards versus memorabilia. They probably should have their own custom tech stacks and solutions, because the products that they offer are different, so it probably means that we need to have almost like an individual technical organization for each one of them. So I decide, you know what, like, it might change, but let me make that the vision, that each business unit has its own technical organization to deliver products and services that it needs to, you know, deliver value to customers. And so … in order....
[00:22:33] Steve Semelsberger
Which might be somewhat counterintuitive to the casual outside listener, like, Collectors is selling stuff online. Certainly everything can be centralized, Dan, but you made a bold bet. So, it sounds like your vision and mission and the process to get there, you were willing to trust instinct, you know, to go against the grain a little bit. That must have required some boldness on your part and on your team's part as well, too.
[00:23:02] Dan Van Tran
For sure, and some easing into it, so …
[00:23:05] Steve Semelsberger
Yeah.
[00:23:06] Dan Van Tran
So the truth is, like, what we ended up starting with is, okay, so what are the steps we need to get there? Oh, we have all this shared infrastructure that we need to build, and all the current technology's crumbling. And there are only so many engineers. So, the first step was actually bring everyone together, have these centralized teams, and work in a central way, and that's fine, as long as everyone kept in mind that one day we're going to start separating out by business lines.
So, that was helpful, to have both the vision, but also the reality of the first step, because then when people were building some of these shared services, like, payments infrastructure or the customer identity platforms. They actually built it in a way where you could actually build unique solutions built on top of shared infrastructure, and that allowed us to not have to rebuild things along the way, because we already kind of bull was, and … That was very helpful.
[00:24:02] Steve Semelsberger
Yeah, that's smart.
[00:24:05] Dan Van Tran
But we started with, you know, hey, we're resource constrained, we are already currently on a monolith, let's just … the first step will have to be from this monolithic, you know, infrastructure with the handful of engineers we have. But then from there, we kept the vision in mind and kept iterating, so now we're about 5 years in, and we're finally at the point where, end of last year, we started splitting the organization into business units. And, just this year, we started dissolving the shared services teams, so that you just whittle it down to the core infrastructure.
And so, for example, for payments, we use Stripe. But the way that everyone uses Stripe is a little different, so we have a infrastructure team that just manages the Stripe integration, and then all the business teams build on top of that their services and APIs and whatever they need, customized for their organization. And so, we had to do a lot, as you might imagine, of infrastructure building in order to get there, but having that vision was very important and crucial in all of us uniting on that path.
[00:25:12] Steve Semelsberger
Has the vision changed during the 5 years?
[00:25:15] Dan Van Tran
I think the vision itself was broad enough where it didn't have to change much. What changed were the milestones in between. As new technologies come, as we learn new things, it just … we had to pivot and adapt, but the way that I view a good vision is almost like a, a GPS, when you are saying, okay, I want to go from you know, New Jersey to New York City. There are a million ways to get there. But the thing that you have to keep in mind is, okay, what … the thing that will stay, similar and the same is that we need to get to New York City.
And so, as you run into roadblocks or, you know, like, there are things that, take a lot of traffic on this road or that road, you can still remember, I need to get to New York City, and then look at different paths with that lens. And so I think, we were fortunate enough to define a vision that was broad enough where we could maintain that and just pivot along the way.
[00:26:14] Steve Semelsberger
It's special, and it sounds like there's a lot of work ahead, and as you continue to look to grow the business, to improve your technology stack, and to operate things in the world that we're a part of right now, with AI creating opportunity and pressure, you know, Dan, I'm nerding out on this intersection of change and resilience, and so I'm curious for you, when you think about, your organization, and the velocity of current change, and how you're updating your production systems.
How do you continue to increase the velocity of change, while also maintaining you know, an adaptable set of capabilities. You know, systems that are modifying that, yet you also feel like you can trust that your team can operate, that your team can go to bed at night, not worry about the 2AM alert. So how are you managing this moment of, you know, balancing risk and speed?
[00:27:17] Dan Van Tran
So, the way that we did it was … You still keep in mind that vision of where the technology should be, and what it'll look like in practice. And as you … focus on building new things, because inevitably, the business leaders, your clients, they don't care what the technology looks like today. They want you to provide certain services, products, etc. So as those requests came in, what we did as a tech team was to look for the overlap. So if you think of it as a VIN diagram, there's the stuff that the customers and business owners are asking for in one circle.
The other circle is the vision of the tech, you know, stack and tech, you know, just the technologies and platforms we want to use. That middle section is what we actually end up, you know, starting with. And so, with each project that came through, we were essentially building the future one step at a time. So, for example, we opened up a lot of, new business lines, new collectible categories that we wanted to get into. So you might think, well, you can just build a brand new stack and just, like, add it that way.
And it would have been faster, for sure, in a lot of ways, but what we ended up doing was asking for a bit more time, but then going into the existing technology and building up some APIs, restructuring some of the databases, just cleaning up house a little bit on that side, and then building the business off of some of those components. And so, even though it would have been easier for us to just build them brand new, we wanted to take the opportunity to migrate some of the old with us. And, that requires, again, keeping in mind what the vision for the future looks like.
And also, making sure that you're delivering the value in a way that your business leaders and customers, they're still getting what they need and want.
[00:29:12] Steve Semelsberger
Appreciate it, appreciate it. Let's … let's continue on this pathway of resilience, and then the things that thwart resilient systems, let's call them incidents. And so, maybe we step out of Collectors, and you can answer this question as you see the industry, as you advise other technology teams and leaders, as you read, as you convene. And so, you know, when I talk to engineering leaders and teams, I frequently hear the era of the simple, single single root cause incident that happened on a golden path. You know, feature flag into production, modern CI/CD pipelines, all that good stuff...and something broke. Like, that doesn't happen so much anymore.
It seems like, there are multiple things happening, you know, that might cause, especially an anomaly or a degradation that occurs over a period of time. So, when things are breaking into production today, what are you seeing some of the best teams and organizations do once they've either received the notification, or the alert, or customer input, or an exec has called them and said something's going on? How are teams responding to these complex incidents today?
[00:30:25] Dan Van Tran
The thing that stays … consistent is that they are very quick to react and figure out, you know, like, hey, like, we need to give a response, we need to figure out what's going on here, and then assess the severity, the priority level for this incident. Is it something we need to fix now, later? How many people are affected? And then that allows you to make the first decision, which is, what do we need to … do we have the time to even investigate this?
Or do we need to just take an emergency step in restoring an old database, switching over to a failover service, whatever it may end up being? And in most cases, that's what you have to do, to buy at least your team a little time to investigate what's going on. But once you have made that decision, then you can start to look at, alright, so what exactly is going on? Because oftentimes, when you get one of these incidents, it is the last step in a much larger process, like you were mentioning. It's a set of complexities that led to this scenario. But all you know is the scenario.
So you have to kind of work your way backwards to see, okay, what are the different ways that might have caused this? and what systems may be affected, so I know who to pull in, or what codebases to look at. And … a lot of that, especially in this era of AI, is very difficult, because the AI might have built a lot of that code for you, and a lot of the specifics around it. So, I have a lot of thoughts around that, but overall, I do think that the human should continue to keep in mind and control the design and architecture, the implementation.
You know, you can delegate some of to AI, but regardless, so, incident comes in, you have to figure out where, what's affected. So, a lot of times, it's important for you to understand those at least high-level components, and how everything fits together. And not … I think at every level, that's important. So even all the way down to a new engineer who's just looking at different codebases and functions, they should understand how those lines of code interact, how they rely on each other, like, what calls what.
The next level up, the manager, owns a set of, you know, areas, code bases, products. They should understand how all those fit together. And then you go up to the director, who understands the entire department, all the way up to me, where if I'm the senior-most leader, I need to understand how all the systems at a high level interact. That way, when something like this happens.
[00:33:02] Dan Van Tran
I know how to address it, even when it's not … you're not talking about an incident, you're talking about a change. I know what is required in order to incorporate that change, etc. So, that is something that is extremely important, and the teams that know this stuff, are very effective in isolating the issue quickly, putting in a quick change in order to restore functionality, or to know what to look at in order to address the issue and find a root cause. And so, you know, sometimes people think. You know?
Like, if we just did more of those tabletop exercises, had better QA, had better plans, like, we'd be able to be on top of all this, the truth is, especially the more complex technology gets, there's only so much that you can achieve with those things, because all of those cover the known knowns, right? And so the things that you are ready and aware of as issues. But with a lot of the ways that things are today, a lot of things are APIs. You have, in the AI world, MCP servers, and all these things in black boxes.
You will not have access to understand how a lot of things are happening in those other systems. So you have to be a bit more adaptable than just preparing fully for specific scenarios. At that level of detail, at least, and just making your team a bit more adaptable to what's coming in, and how to just, for any sort of generic situation. How to quickly triage it and, remediate it.
[00:34:36] Steve Semelsberger
Yes, we hear repeatedly, stories that sound like it's the code that goes into production, it's the unexpected traffic pattern that emerges, it's the degradation from the vendor that you have a dependency upon, and then it's a human config change with good intent. And so, what's the root cause? There isn't one. There's four contributing factors, and figuring out how those intersect to get to the situation to then determine the repair path, is, you know, either where the pain or the fun begins, depending upon your tolerance and how you look at it.
And it kind of leads me to this moment in the industry. We were speaking with a Gartner analyst a couple weeks ago, and he's challenging his clients to spend 5-10% of their technology team's time on preventative work. Not just, like, basic maintenance and upgrades and improvements, but, you know, really being thoughtful about the containability, the malleability, the adaptability of their systems, holistically, not just the code. Is prevention something that you see at your clients, companies you work with, Collectors? Are you seeing a trend yet around prevention, or is this more an aspirational analyst challenge?
[00:35:52] Dan Van Tran
No, I think that all the best organizations Have at least some level of prevention management. You know, as … The adage goes, prevention is the best medicine, right? Like, the best way to handle an incident is to not have the incident to begin with. So, I think that it's very important for people to constantly know the limitations of their technology, where it's brittle, and Tried to figure out ways to … prevent issues from happening there. And so, in some cases, it's just being proactive about refactoring and building.
But a lot of times, it really is just figuring out the prioritization of certain things. Because there really is, in some of these cases, not everything's a risk in life, right? And, like, and especially in code, the second you commit that change, it becomes a potential issue. It's already a legacy debt. So, you can't possibly rewrite your entire application tech stack, you know, over and over. And so what you need to do is just treat it like a feature. Treat it like these feature requests you're getting from your business leaders and from customers.
Look at the risks of your system, think about the impact, the severity, and then weigh them against the other things, and prioritize them appropriately. Because not everything needs to be repaired or fixed. There are some … a lot of times when putting a band-aid on something takes very little effort, and it's fast enough, doesn't really impact too many people.
But if you were to change that technology, it's a huge undertaking, it would involve tons of people, lots of time, and that's time that is an opportunity cost, because you could be building new functionality, or doing other things with that time. So … Yeah, so prevention definitely is … an important part of every team's, incident management. It's the proactive prevention of incidents to begin with. So … I think that it is important for every Technologists to understand where the gaps are within their own technology.
To weigh that against all of the other requests that are coming in, in terms of features or other bug fixes, because a lot of times, we treat features as the most important thing. But sometimes it gets to a point where there's so much technical debt that's accrued that we need to Stand up and actually push against those requests, and ask that we spend time to remediate an issue, because… as much as the business owners and the customers want that new feature, what they would, hate is if that feature came at the cost of not being able to use the product or service at all for a period of time. So … We, as technologists, again, it's important for us to be aware of all those things, and weigh them against the queue of work, and prioritize things appropriately, and when it makes sense, to request the resources and time in order to remediate those issues.
[00:39:04] Steve Semelsberger
Yeah, well put, well put. You know, there are various attempts to estimate the impact of an hour of downtime on businesses in different industries of different sizes. Do you find that most businesses are able to really estimate their own individual minutes or hours of downtime impact? And if so, are they able to use that, you know, in partnership between, say, the CFO and the CTO to justify prevention work, or is it more like, finger-in-the-wind estimate, you know, nobody really knows how expensive this is. We just know it hurts our brand, it hurts direct revenue, hurts our place in the market, it hurts trust with our customers. Talk to me a little bit about what you're seeing, if anything, around the ROI of prevention.
[00:39:52] Dan Van Tran
Yeah, I think, for me … the answer is, it depends. It depends on the situation. And so, there are some cases where downtime is important. So, for example, in some companies, downtime means that the customers cannot, work with your products and services, right? If Amazon goes down, no one's buying anything if the website's inaccessible, so that's a huge … it's almost impossible to measure exactly, but you know that that's a huge problem, and so you try to prevent things like that.
There are other cases where it's some back-end infrastructure, and so, okay, so the internal employees can't work for a period of time, you know, like, how big of a … an impact is that? So, a lot of times, instead of … creating exact numbers, and actually, we use a platform called Incident.io, so we know the exact time that each one of these incidents, you know, like, for each stage of it, even, how much time we're spending with the system being down, or remediating, or how long it took to bring it back up. But none of that actually is all that important.
It's helpful for some things, for some efforts in improving the process, but Overall, we just look at severity on one axis and impact on the other. So it's a typical risk matrix. And, so then you can actually look to see, like, well, how many people are impacted, and how are they impacted? And come up with a quick assessment of, is this low, medium, high, critical? and treated accordingly. So, I think that … Yeah, so if you go to the CFO, it's pretty clear to them the different magnitudes of those two issues.
One where customers are unable … to access your services for an unknown period of time, versus an internal employee not being able to do their job for a period of time. They're both impactful, you can measure both, but they don't really need to be measured in order to get the attention to, in time to fix the issues.
[00:41:59] Steve Semelsberger
Yeah. Yeah, I find that, you know, when the CEO and the CFO have a sense of the dollar impact, it's easier to look at things, you know, almost like preventative measures as the gym membership or the health insurance plan, you know, you put some money into it, you know, because you're hoping for the absence of the pain. And it can be hard to quantify this notion of, you know, the ROI of absence, because you do great preventative work and the thing doesn't happen, and does that budget get cut in the future? You know, or do you continue to maintain it?
These are … these are tricky things that I think the industry as a whole is working on. There's a group called the Resilience in Software Foundation that's led by John Allspaw, who is the former CTO of Etsy, and they're doing some really good and deep work in areas like this.
[00:42:51] Dan Van Tran
Yeah, I mean, some rudimentary things that we've used in the past are things like incident rates, you know, regressions, and things like that, and then you could Overlay on that, the cost of … How much are you paying employees, and … How much do you generally bring in per hour, or per day on these systems? And so you can get some rough metrics, but it would be nice if a lot of that were automated for us.
[00:43:18] Steve Semelsberger
It's tricky, it's tricky, but there is opportunity. And so, as we come to a close, we talked about, your background, your trajectory, your journey at collectors, you know, how you're thinking about growing, transforming, operating, preventing.
So, you know, we're in this world, and I know you're thinking a whole lot about the intersection of humans and AI, so I wonder if you put your futurist hat on, and you imagine an engineering team 5 to 7 years from now, and, you know, there are agents writing code, reviewing code, deploying software, you know, watching telemetry, diagnosing incidents like we've talked about, proposing fixes, maybe making updates, you know, full SDLC, dreamy kind of future magic world. So what is … what does the human engineer do in a world like this? We can debate whether or not that will actually exist, we all have opinions on the pieces of it, but pretend it does. You know, what's the role of the software developer slash engineer?
[00:44:24] Dan Van Tran
I … think about this often, mostly because I want to prepare the engineers on my teams, and also I volunteer at a community college, I talk a lot to vocational schools, I want them all to be prepared and build the right skills for that new world, which is coming very quickly.
[00:44:44] Steve Semelsberger
Very fast.
[00:44:45] Dan Van Tran
Very fast. And at this point, it's very realistic for any organization to have an agent cloud of developers that each developer themselves are using to actually do the coding. So then even the entry-level engineer needs to just think more about design, and a lot of, like, how things fit together. Which they're not taught. A lot of us, we learned that through years of experience, and now suddenly we're in a world where we need to train the younger generation on how to do that immediately.
So, if you fast forward, I think that a lot of, university programs and degrees, like computer science degrees, will change, unless you are going into academia. And, you'll have much more practical, like, education on how to manage agents, how to work with, you know, and you'll be coding through agents. And it's almost like, back in the day, I learned C++, and that was the language that all major colleges used. And I love C++, by the way, it's still my favorite, but you had to do the memory management and all the performance and efficiency gains all manually.
And then later on, there was …
[00:45:51] Steve Semelsberger
In my first job, Wall Street 150 Broadway, we were working with some of the original C++ compiler folks out of Bell Labs. We were kind of at this intersection of research and some of the first three-tier fixed income and equity trading systems on Wall Street. So I go back there with you, Dan, on the days of C++, but keep going.
[00:46:12] Dan Van Tran
Yeah, But so you go … one generation past that, maybe, like, 5 or 10 years, and then everyone's learning Python, which is, like, just … it's very cool, and a very nice language, but you don't have to know as much of those, like, core competency, like, things you were learning in school, and … Now you fast-forward to today, and even with a lot of, like, the React frameworks, and a lot of the Flutter, and all these, like, front-end libraries, you don't even need to know, a lot of times, like, the core concepts of programming, period. Just how to use these widgets.
So … Especially with AI, AI can now do almost all of it for you in terms of the coding. And that means that the students, the education, everything needs to evolve to better match that if we want to, you know, educate people, prepare them for a world where we exist with AI. So, going back to your original question, if we fast forward to a point when all this is in place, I think that the teams are much smaller.
I think that things like Agile actually … principles of it stay, but most of it goes away, because you no longer need to match tickets, you're now managing epics, if not something larger. You still need project management, but it's a far different type of project management.
It's now, thinking about much larger components And, a lot of the typical roles, like, I think about product managers, you likely will have product managers in some scenarios, but a lot of where they exist today, you may not need them anymore, because you are able to take the shortest path of going from an idea that some business leader has, and then using the AI to implement it. And the AI will have guardrails and everything else built in. You may need an engineer there for a period of time, and maybe, you know, until we get to quantum or something, but you only need maybe one.
And you don't need them to help, the business … you don't need a product manager to help the business figure out what products and services to build, because AI can help them. The AI will be able to figure it out. And so … the truth is, a lot of business leaders may not want to do that sort of work, and so that's where a product manager may fit in. Also, you have a lot of product managers who are pretty technical, and they'll probably have a lot of fun building the POCs and actually implementing the technology itself. But all these roles evolve.
I don't think there will be a single role in tech that isn't touched in some way. And the thing is, we all shift left, right? To borrow a term from QA, we all shift left. And so, even, you know, but the leaders under me, they become essentially, like, CTOs, making decisions for their entire organization at a much higher level than they were historically. The leaders under them are making architectural decisions that they never used to have to make, and on and on.
[00:49:00] Steve Semelsberger
It's such a fascinating moment, you know, as you and I have been in industry for lengthy periods of time. You know, I've seen lots of waves of automation and technological improvements, you know, and there's been an ongoing question of how many software engineers do we need on the planet? And with each wave, there have been more. And some of the data that I've seen recently, we're recording on August 25th, 2026, is that there are a lot of engineering jobs open, and it's team dynamics and … work structures and ways of working and places of working are shifting.
There still is a lot of software being built, modified, improved, strengthened, hardened, made more malleable, made more adaptable. So, you know, I've been interviewing some futurists with this podcast. You'll hear from Brett Hurt and Byron Reese, and they are all in on a coming age of abundance for the entire planet, the ecosystems, nature, humans, the whole thing. And the bet is that software will continue to be more, and that there will be lots and lots of room for people who want to do work, yet the ways of working, the quantity and quality, of work will continue to shift.
And so, you know, one piece of guidance that I'm giving to those, earlier in their career is to work hard on the human-to-human piece, because, you know, maybe 25 years ago, there was tolerance for not-so-nice, technical folks who would sit in the corner, and, you know, they knew the systems, but they were really hard to work with for other people, and I think, you know, part of the gift of this moment is we can all become a bit kinder, you know, we can change in ways that, you know, are more full of gratitude, joy, other centricity, and things like that.
And so maybe that's a place for us to end, Dan, as you think about your personal journey of growth, you know, as you look at leadership, as you look at, you know, how you show up in the world, what are some of the things that you're working on changing and growing and evolving into?
[00:51:10] Dan Van Tran
Yeah, as part of, is … it happened before this AI evolution, actually. I … Always wanted technologists to be closer to the problems they were solving, and understand them better, so that they can propose better solutions. But so, I agree with you about that future world. I think that … it's amazing that the technology's finally accessible to people who never really would have been able to use it to begin with, people who aren't able to code. It lowers the friction of people being an entrepreneur, coming up with an idea, solving a problem in the world.
And so, to the point about abundance, it's giving a lot of people the ability to solve the problems and create this abundance for us all. And so … The part that is important, though, that everyone needs to learn, is the entrepreneurship. And I do feel like Some people say, oh, it's just something that, you know, people have, an inherent skill, it's hard to teach. That's true in some cases, but I think that you can teach enough for someone who doesn't feel comfortable working in that way, and give them enough tools to be able to do that.
And I think that we should do that to see what … how people contribute to society and the world. Like, what are the positive things that they're able to introduce because they are able to take a problem and solve it in their own unique way.
[00:52:31] Steve Semelsberger
Yeah, great, great, and I definitely agree with you. Anything you'd like to share about your personal journey? Was that a hint on your own journey of entrepreneurship? Or, you know, I know you have some chickens, you know, I know you have a young family, so, I mean, I imagine you're learning and growing all the time, so maybe as you look ahead over the course of the next year, is there anything in particular you'd like to learn, see, experience for yourself?
[00:52:57] Dan Van Tran
Over the next year, I think one of the things that I am looking for is the opportunity to work with a lot of organizations who are stuck And not sure how to move forward into this world. And help them to navigate that path, build the bridge for them, give them some ideas, guidance. And so, I think that it's, It's … the growth for me is keeping up with all this change that's happening, and so I actually created an army of agents that go through and parse multiple times a day all the news and trends and suggest articles to me, and it's helping.
It's actually helping me to keep up with a lot of things. But the truth is, like, especially with AI, a lot more content's being created than, you know, like, our ability to ingest it. And so … over time, I'm gonna have to teach the AI how to prioritize better for things that are more important to me and all that, but I think that it's, keeping up with all this is a challenge that I enjoy, but a challenge nonetheless.
[00:53:57] Steve Semelsberger
Well, I know that people can follow you on LinkedIn, and that you publish some very thoughtful perspectives on a broad variety of topics. You're a very special human being, you're generous, you support entrepreneurs, you support technology leaders, you support businesses, you support communities. Your personal story is one that I admire, as you've shared the journey of your family and how that's impacted you, and I think you are a wonderful example of somebody who is Change Resilient, which is why I wanted you to be one of the first guests on this podcast. So, Dan Van Tran, thank you so much for joining us today on the Change Resilient podcast.
[00:54:35] Dan Van Tran
Thank you so much, very humbling.