Few engineering leaders have shaped globally transformative tech products quite like John Thimsen. John’s journey began at age 12 where he taught himself how to code, laying the foundation for his pursuit of a degree in Computer Engineering at the University of Washington. Following university, John joined Microsoft in the late 1990s to work on Internet Explorer, and then joined Amazon, where he became the fifth employee on Project D — the initiative that later became Alexa. As Chief Technology Officer of Qualtrics from 2015 to 2022, John scaled the engineering organization from 50 people to over 1,500 while helping drive the company’s growth through its successful initial public offering (IPO).

Crew Capital recently hosted a roundtable with founders and engineering leaders from our portfolio companies where John shared insights on building engineering culture, managing team transitions, and evolving as a technical leader. Among the many highlights, John detailed how engineering leaders can create high-performing teams while avoiding common pitfalls that emerge amid periods of rapid growth. From establishing essential processes with a five-person team to managing a thousand-person engineering organization, John offered a comprehensive framework for technical leadership across all company stages.

Building strong engineering foundations

Early in the roundtable, John emphasized the critical importance of being deliberate about engineering culture. “One of the key things I learned at Qualtrics is the need to be intentional about the values that you want to cultivate within your team,” John said. “Being thoughtful about what you want your teams to value both today and in the future are key items to address from day 1.” At Qualtrics, these values included learning through failure, ownership of outcomes, transparency, and mastery of craft – But John stressed that having values alone wasn’t enough, leaders need processes to reinforce those values. 

“Every single all-hands I ever did with the engineering team, I would put these values up on a slide and I would talk about them, one by one,” he said.” I would have explicit mechanisms in the organization – Root cause analysis meetings are a good example, where we would talk about failures in order to learn what caused them and how to avoid repeating them.”

For early-stage startups, John advocated for keeping processes lightweight. “When you have fewer than five developers, you’re still at a team size and shape where you can easily manage the complexity of a roadmap or a plan using a shared whiteboard or a shared wiki page,” John said. He recommended three essential processes: Weekly planning meetings to set goals, regular demos to showcase progress, and thoughtful recruiting processes, as early hires make the greatest impact on the company’s trajectory. 

On the technical side, John reiterated early investment in quality and testing. “It’s a false trade-off,” regarding teams that skip testing early on. “It feels like you’re moving faster, but you’ve all seen the charts that show you how expensive it is to fix a bug depending on where you find it. If you find it on the developer’s desktop, it can be fixed in minutes, but if you find it in production with the customer, you’ve got a week or more of work.”

The 5 critical engineering team transitions

Through years of scaling engineering organizations, John has identified five distinct transition phases in the journey of team growth.  

The first phase occurs with teams of fewer than 20 engineers, where project-based execution is the norm. “You’ve got a capable co-founder and maybe one other trusted tech leader and manager. You’ve got a strong project focus,” John said. “It’s perfect, It’s what you want. You’re on that project, get it done by next month, and you sort of rinse and repeat to achieve these milestones that are critical when you’re still young.”

This project-focused approach hits its limits at the second phase, at around 25 to 30 engineers. “You start to have this notion of a durable team with persistent ownership over a part of the product or platform because you have to establish a scaffolding for how you bring engineers into the organization,” John said. “A project-by-project based approach won’t scale, but a functional organization where you’ve got teams with distinct identity, and you can add engineers on teams that own hardened parts of your product, will scale.”

The third phase comes at 100 engineers. “At that point, you need to start building out senior manager and director functions,” John said. “This is where it becomes critical to have organization-wide standards for quality and availability,” John emphasized the importance of measuring output metrics at this stage, tracking everything from regression rates to feature delivery velocity.

Organizations typically transition to multi-site operations at the fourth phase, with around 300 engineers. “At this point, you’re building out the director and VP functions. You’ve got multiple dev centers,” John said. “You’ve got mature annual planning processes and budget allocation in place, and your span of control is defined by ratios.”

The final transition occurs around 500 engineers. “Past 500, it’s more or less the same, but you’ve got SVPs reporting to you,” John said. “As a CTO at this scale, you should be able to hand an SVP a business outcome that you want to be affected – go solve information security, go build this product. That’s the altitude you’ve got to be able to operate at.”

Navigating enterprise customer relationships

Early-stage startups often face skepticism from enterprise customers about their maturity and capabilities — particularly around security and compliance. John’s experience at Qualtrics offered valuable lessons in addressing these concerns.

“The role of the CTO is pretty critical early on,” John said. “The CTO needs to be very comfortable having conversations with customers around data security, privacy, and compliance protocols and certifications – that was a key role that I played early on in Qualtrics.”

When dealing with large financial institutions or government agencies, John emphasized transparency over pretense. “We had a lot of government deals that hinged on FedRAMP approval. Now, we also had a lot of commercial companies who wanted us to achieve FedRAMP just because as a proxy, it indicates that the business is thinking about information security in the right way,” John said. “A lot of the time, it really boiled down to having a detailed understanding of what the standards were, why they mattered, and what the road map was for delivering on them when talking with chief information security officers (CISO’s) of customers who wanted to bring us on board.”

This approach of open communication proved effective even when Qualtrics didn’t have every certification a customer wanted. “It’s a risk assessment,” John said. “If you look at customers who are doing security assessments of their vendors, it’s not uncommon for them to have vendors that aren’t checking all the boxes — probably more common than not. Then, it’s really about how that vendor is responding. Are they trying to obfuscate and say, ‘We only have what matters,’ or are they being honest and transparent about where they’re at today, and what the roadmap is for achieving full compliance?”

The evolving role of technical leadership

The challenges a CTO faces change dramatically at every step from seed stage to IPO, requiring constant reinvention of the role. John experienced this evolution firsthand at Qualtrics, where he joined a company that was under-invested in Research and Development (R&D). Almost immediately, John had to put out a huge fire.

“When I walked in the door, we had just begun to split apart a PHP monolith – a massive, single application where all functionalities lived in a single code base. We had no centralized QA team. We had no Information Security team. We faced significant quality issues on almost a daily basis,” John said. “We had significant availability issues, including multi-day incidents that were customer impacting.”

Faced with these obstacles from day 1, John had to ruthlessly prioritize his focus. “The primary challenge that this business had was we didn’t have enough engineers by probably close to an order of magnitude,” he said. “And how does the R&D function play a role in helping drive top-line growth? It’s new products, capabilities, and renewals.”

This realization led John to focus intensely on hiring, spending more than half his time building out the recruiting function in his first year. “We hired 50 people in the first 6 months that I was there,” he said. “I spent a majority of my time recruiting and working closely with the HR organization to operationalize hiring.”

For other critical needs, John’s strategy was to delegate through strategic hiring. “I hired a head of Quality as one of my first hires. It was an important enough challenge that we needed to go in and solve it, but I wasn’t going to be able to do it on my own. We had no Information Security function, so I also quickly hired an information security leader,” he said.

The cornerstones of healthy engineering cultures

One of John’s final points centered on how engineering culture takes its cues from the foundations laid by leadership. “Culture is emergent,” John said. “It’s a result of the norms that are established for how you work together. A lot of culture is reflected from the top, like how your senior leadership team works together in solving company-wide problems.”

He found that sharing both successes and failures reinforced the values he wanted to instill. “What was interesting for me is that every time I would do an all-hands, I would share success stories for the business, but I’d also share where we messed up as an organization and what we’re doing to respond,” he said. “I would share situations where I personally messed up and what I’m doing to learn about it and adapt.”

This commitment to transparency and continuous improvement became a cornerstone of Qualtrics’ engineering culture. “Being super thoughtful about the values you want to focus on and how you reinforce them in the ways that you work is the easiest way to affect culture over the long-term,” John said.