Platform Engineering 101

Platform Engineering 101: Takeaways from Building Internal Tools at HubSpot - Interview with Zoe Sobin (Senior Engineering Manager, HubSpot)

Platform engineering is far more different from product engineering than you might think at first glance.

Building internal tools means that you have internal customers, but you also have external customers as well. This requires different communication methods.

You need to support platform tools in the long run. This means you need to make them extremely flexible, durable and reliable.

We bring you a case study for figuring out these challenges around platform engineering from Zoe Sobin, Senior Engineering Manager at HubSpot. This blog post is written based on episode 61 of the Level-up Engineering podcast hosted by Karolina Toth.

This post covers:

• Platform engineering at HubSpot
• Signs you need platform engineering
  • Customer feedback
  • Evolution into platform
  • The rule of three
• Getting buy-in for a platform product
  • Buy-in from leadership
  • Internal pushback
  • Buy-in from product teams
• Communication channels in platform engineering
  • Early stage communication
  • Current communication
• Prioritization in platform engineering
  • Learn to negotiate
  • Effort vs impact
  • Utilize product managers
  • Use data
• Skills for platform engineers
  • Patience
  • Communication
• About Zoe Sobin

What’s your platform engineering team doing at HubSpot?

My team is working on a centralized reporting tool for the engineering teams at HubSpot.

HubSpot is basically a CRM, with businesses using it. We help them send marketing emails, close sales deals, and support their customers. Reporting is necessary for them to see the impact of their actions, so we collect their data and reports in one place.

When I started working on reporting, we had different solutions built for different reports. For example, email reporting and deal reporting had different point solutions.

My current team is building HubSpot’s reporting platform, including dashboards, libraries, and report builders. As we keep working on the platform, it continuously supports more types of reports. We’re planning to scale it further to support more types of data and more report interactions that have been requested by product teams.

In our platform engineering team, we have a front-end and a back-end team. The teams are structured in a similar way to most of HubSpot’s product teams. Every team has a group of software engineers and a team lead, and each group is partnered with a product manager and designer.

The processes are subject to change. They evolve with the company and the people who come and go bringing their own styles to the table.

How did you realize that you needed a platform engineering team?

Customer feedback

We’d been receiving feedback from customers about our teams using different ways for different reports, like line charts, bar charts and tables. Different teams made different decisions about presenting reports, which led to confusion among our customers because reporting wasn’t consistent across HubSpot.

Building a platform solution benefits both our customers and our engineering teams. Our customers gain clarity from our reports becoming consistent, and our product engineering teams don’t have to build reports for themselves.

Evolution into platform

The original idea for our project was to have one place for all the core reports. It didn’t start out as a platform engineering project; we just realized along the way that we couldn’t easily plug other reports into one place. We needed to build an end-to-end tool, and we had an opportunity to turn it into a centralized platform that scaled across data types, report types and teams.

We built a core reporting platform first, then we worked on our own customer facing applications on top of it. Afterwards, we started talking to teams that had point solutions for reporting about getting these supported by our infrastructure.

The difficulty of building a tool for everyone is that it won’t be perfect for anyone. For example, the email team knows what type of reports their customers prefer, and the centralized tool may not fit their needs. In platform engineering, you have to compromise on some functions and build a central solution that works for 80-90% of use cases.

The rule of three

Software engineers are taught to avoid duplicating code, so it’s always tempting to abstract your code and turn it into a platform solution. The problem is that you don’t have an understanding of all the use cases, and your code may not be the best way to deal with those. When you promise a solution to other teams, you need to run it as long as they want to use it, and they will keep coming to you with feedback and requests.

My manager taught me the rule of three. It’s a software engineering principle: Don’t be afraid to build something three times before you abstract it. This is great advice for platform engineering.

This makes you think through numerous use cases, helping you solve many issues that you didn’t see coming. This approach makes you build better abstracted solutions.

How did you get buy-in for your platform solution?

Buy-in from leadership

I recommend you to be careful when you start thinking about turning your project into a platform solution and support other teams in your company as a side project for your team. As soon as you make that commitment, the requests start to snowball. Make sure that supporting your platform lines up with your team’s vision and resources, or your team will be overextended in no time.

This is why we needed buy-in from management. We went to our manager and explained our idea and how it can be valuable for the customers and for the business. It was an easy sell because our engineering leaders have a technical background and they’re close to the product, so the clear problem-solution relation resonated with them.

Internal pushback

Missing features

We had objections from product owners, who were pushing for certain features but couldn’t get them supported. Some pushed for features that were doable, but the quality of the data they collected wasn’t good, so it turned into a request to change the quality of their data. These may lead to objections about the platform not measuring up to expectations, but you have to stick to a set of requirements to build flexible solutions.

Questioning the platform’s necessity

When people see complaints and issues connected to platform engineering, like certain teams lacking the support they want, they may assume that the platform isn’t working. These objections tend to come from less technical people.

Manage expectations

You have to balance between selling your vision and being transparent about the reality of building a centralized platform tool that may not solve every use case from the start. You want to get everyone excited about your platform product so they commit to it. At the same time, you also need to figure out ways to help everyone through the reality of achieving that vision.

Other platform engineering teams in HubSpot have been struggling with this as well. Building and running a platform is a multi-year effort and the expectations can get frustrating. The best you can do is find compromises that you can build and maintain and that work for everyone.

Buy-in from product teams

Platform engineers always aspire to build a perfect tool that serves every team’s needs. We went to talk to the other teams that use reports with that attitude to sell them on the vision, and they came on board.

On one hand, they understood the value our team could provide, but on the other hand, it made them nervous. They care about certain features that they use for their customers, and centralized reporting could have taken some of those away. In order to get buy-in from all of them, we invested a lot of time into solving these needs and convincing these teams to adopt our solution.

In retrospect, this may not have been the best way to get buy-in.

Working with teams who were interested in our platform solution and didn’t want custom features for themselves may have been a better use of our time in the early stage. You may have to walk back on some custom features you promised in the early days, or delay them as you start dealing with the technical implications of these commitments. It’s also easier to get support from teams that don’t require special features, so you can focus on delivering the core platform for them and deal with special requests later.

Despite this, I don’t regret taking this direction. We managed to get everyone on board, and we made the tool work.

What’s the communication process between platform engineering and other teams?

Early stage communication

Direct negotiations

In the early stages of setting up the reporting platform, we spent a lot of time negotiating with the teams we were building the tool for. We spent even more time with the engineers who had strong opinions of how they wanted reporting to work. My team and I traveled to Dublin to have in-person meetings with engineering teams in the Dublin office and to understand their needs and reservations.

Product roadmap

After the initial conversations, we broke down their requests, committed to building some of them, set up a priority, and turned it into a roadmap that we shared with all the stakeholders.

We did this five years ago, and HubSpot has grown a lot since then, so this may not work for us today. Having lengthy negotiations with every team using our platform could take too much time.

Current stage communication

Centralized channels

Currently, we rely on a ticket-based system; we also use a centralized Slack channel where people can ask questions and make requests. Product and UX lends a hand with vetting those requests, understanding where they’re coming from, and how we should prioritize them. Based on what we’re seeing in these channels, we may escalate certain requests to a meeting.

Direct communication

Despite using centralized channels, we can’t stop talking to our customers. We have to stay connected and understand the needs of the product engineering teams. If we ever stopped collaborating with our opinionated engineers across HubSpot, we’d soon end up building the wrong features.

Build transparency

We hold weekly tech talks at HubSpot, where the engineering teams get together and someone gives a talk on a technical topic. I’ve presented about the inner workings of the reporting platform a few times. This helps us create transparency with the way we work, and it helps other engineers learn from our experiences.

Find the source problems

Product engineering teams often don’t know how to engage with platform engineering. You need to show them the best way to get results from a platform team.

In a product team, you’ll have an easier time approaching a platform team with your problems and your customers’ problems. This allows you to partner up to find the best solution together.

When you go to a platform team asking for specific solutions and features, you may have a difficult time, or you might just get a flat no. Take the time and be vulnerable, even if you think you have the answer.

How do you prioritize requests in platform engineering?

Learn to negotiate

You need to learn to negotiate in platform engineering. You can’t say yes to everything, but you can’t always say no either. Most of the time, you need to negotiate a compromise between requests and what’s possible or what makes sense for your team to build.

I had a hard time learning this; my manager coached me through it. I was freaking out over requests that seemed outlandish from my team’s perspective. He told me that I can’t just turn them all down; I need to show them my point of view because they lack that insight, and work with them to find a compromise.

Measure effort versus impact

We navigate prioritizing requests based on effort versus impact. The idea is to prioritize low effort and high impact requests to provide the best value for our customers.

In platform engineering, your customers are other engineering teams and the end users, so you have to balance the impact on each side. Greasing a squeaky wheel in engineering teams can have a high impact on the end user. It’s up to your team to examine all streams of customer and team feedback to figure out what’s going to have the highest ROI.

Utilize product managers

Prioritization comes down to knowing your customers and working with your product managers. Some guesswork will always be involved, so the best you can do is utilize data to come to the right decisions.

When teams make requests, they often don’t know how difficult these will be to implement. At times, they make requests like customizable colors on certain charts that are already on the roadmap, so we can just let them know it’s coming. At other times, they request for a complicated custom interaction, and we have to explain that it’d take a lot of work for just one use case, and we don’t have the bandwidth to prioritize it in the foreseeable future.

Platform engineering has to navigate different tiers of requests, prioritize the work, and set expectations with the stakeholders.

Use data for conflicting requests

Product managers play the biggest role in platform engineering when you receive conflicting requests. I always aim to involve engineers in these conversations, but it usually comes down to prioritizing and making tough calls. Team leaders may have an opinion about these requests, but they should consult with their product manager.

Product managers can bring a perspective based on feedback from end users, and they can collect more data to back up a decision like that. They’re familiar with relevant metrics we’re tracking and previous experiments that we can learn from. They know what data to check to find the best path.

What skills do developers need in platform engineering?

Product versus platform engineering

Platform engineering deals with different problems compared to product engineering, and not every developer is interested in that.

Platform teams have to think about problems in an abstract way. You can’t zero in on solving one thing; you have to build a tool for a wide range of use cases. You also have to look forward, consider maintenance over the long run and how your tool might add technical debt to future development.

At the end of the day, our team needs engineers who are interested in business intelligence and data visualization, on top of building cross-cutting platform features.

Patience

Developers in platform engineering have to take their time to write clean code and do everything the right way. The platform has a huge impact, so mistakes are costly as well. We need people who are comfortable working this way.

In platform engineering, you’re dealing with hard problems that take a long time to solve, and the work is highly detail-oriented. When you see platform engineers lose their motivation, you have to help them unpack the reasons behind it and try to find a solution to their problem.

Strong communication

Communication skills are important, because we need to collaborate with many other teams at the same time. We collect requests from other teams, help them understand what we can or can’t do, and the reasons behind that. Platform engineers need to be able to negotiate, because we may be able to do some requests in a given time, but not everything, so they need to help other teams prioritize or offer alternatives.

I love platform engineering. It’s a challenge on both the technical and the people side. Sometimes, I think everyone must want to do this kind of engineering, but it isn’t for everyone.

About Zoe Sobin

Zoe Sobin is living in Boston, Massachusetts, close to the ocean and some mountains. She enjoys spending time with her husband and dog and going out into nature.

Zoe is currently a Senior Engineering Manager at HubSpot, where she’s been working for seven years. She started her career at HubSpot as an intern, and she started working there full-time after graduation. She’s spent most of that time on the reporting team and she’s been growing with the product and the company over the years.