Podcast Special
Are you a data enthusiast who is excited by the synergy of reverse ETL, curated metrics, and AI's potential to revolutionize data-driven decision-making? This one’s for you.
Allan Wille, CEO at Klipfolio, sat down with Brian Kotlyar, VP of Marketing & Growth at Hightouch, a reverse ETL solution that makes data manipulation, extraction, and moving smooth, easy, and fun for data engineers. Their conversation explored the potential of reverse ETL, the pressing challenges of modern business intelligence (BI), and the exciting prospects that are emerging as AI meets data analytics.
Note: This article is a shortened and edited version of a podcast, highlighting the most important insights. Listen to the entire conversation on the Metric Stack podcast.
Reverse ETL with Hightouch
Reverse ETL enhances the way data engineers unlock value from raw data by making data manipulation, extraction, and movement seamless. It’s about ensuring that the data is not just collected, but also refined and made ready for value activation. The process bridges the gap between raw data and actionable insights, which is crucial for businesses aiming to make informed decisions.
Allan from Klipfolio: Brian, tell the audience about Hightouch and reverse ETL. What is reverse ETL?
Brian from Hightouch: Reverse ETL solves a problem that has plagued me for the entirety of my career. We pour all this effort into gathering data and processing data to understand our business and our customers. But, we consistently run into this brick wall of actually using the stuff. Reverse ETL makes using data day to day, to send a good email, to provide the appropriate offer to a potential customer, to customize an experience on a website, or in an ad possible.
Allan from Klipfolio: I know you've been thinking about the “value activation” idea for quite some time. Tell me a little bit more about that. Where are the pitfalls? What are some of the ugly things that you've seen?
Brian from Hightouch: There's all this work that actually represents probably 75% of what we do, which is extracting lists, passing them around, and getting them back. Could you add a cut of this? Could I get a pull of that? This never-ending list of requests and little tidbits are not exactly what we sign up for.
The thing is, it’s a real part of the value that analytics teams create, though it's not in the job description. Typically, it's often not the thing that is perceived to be the highest order work. The reason we're getting all these requests is they create a ton of value. This thing that you feel as a source of frustration and not the fulfilling part of your job could be the most fulfilling, the most exciting part of your job because you're actually helping a line of business people with urgent requirements.
Allan from Klipfolio: Give me an example and let’s paint the picture of a data engineer signing up with Hightouch. What aspect in their work lives will be improving, making them happier?
Brian from Hightouch: There are two classic scenarios, but they’re very similar. The first is, when code for data extraction has been written by someone who now works on a different team or even a different company. No one really understands it. It gets out of date but it doesn’t break very often. So, it’s fine. Every company has this ocean of tech debt but it is nobody’s core role to build and maintain these pipelines. Until reverse ETL existed, there was no proper solution where all of this could live. With really high throughput synching, Hightouch ensures great data capture and excellent observability. We can manage it by API or by git sync or however we want to do it. They can trust it is going to be a bulletproof piece of infrastructure that will always work. And if it didn’t, you would know why.
The second classic scenario is a little more subtle. It happens when somebody doesn’t even want to take the time to do these things. For example, they don't want to generate the CSV to download and upload somewhere the usage is throttled. The actual application of data where it would be needed, for example in HubSpot or Zendesk, is extremely reduced. Moving data with a tool like ours minimizes the effort to essentially zero. There is no incremental work required to move it to HubSpot and Facebook or HubSpot and Facebook.
The best part is, it’s good data. It’s observed, clean, controlled data that the analytics team can work with. This “golden data” is being populated in a controlled way everywhere, all the time with no extra work. This way, you get a data-driven business as a result.
Challenges of BI
Today, one of the prominent challenges in business intelligence is the fragmentation and complexity of data sources, leading to a lack of standardized and curated metrics. This results in discrepancies across various reports and dashboards, hindering confident decision making. To address this, businesses need to centralize their data into a single source of truth, maintaining consistent definitions and context. A modern data stack, including a semantic layer, can play a pivotal role in achieving this, ensuring data quality and accuracy and enabling meaningful insights across the organization.
Allan from Klipfolio: There are a number of reasons why BI goes off the rails. For example, the data is not rich and structured enough to actually make an informed decision. Or, there are 15 reports that are 4% different from each other. You're talking about creating this single source of maintained truth that the data team now owns. What does a single source of truth mean to you guys?
Brian from Hightouch: I think we are actually the beneficiaries of an ongoing movement in our industry to try to consolidate and centralize a lot of this stuff. What we've seen is that as companies move towards software to do this they tend to consolidate data into one cloud data warehouse environment, which can actually be very complex. But at least it's all in one spot.
The technology has gotten really good. We just come in as a beneficiary of that. If a company does all the work to standardize definitions and consolidates all their data into one cloud provider, they can say “Well, I have two mandates: business intelligence and activation. I’m doing pretty well on the business intelligence side. The activation, however, is still happening in spreadsheets and custom scripts. Let me also adopt tech to make that work well and be scalable.”
Suddenly, the data team would live up to its potential because they're solving all the BI requests, they're keeping their CFO in the know, and they're supporting their business stakeholders with consistent dashboarding and consistent data definitions. And, in parallel, they're also feeding all these operational one-offs in a very scalable way.
Allan from Klipfolio: Brian, you previously mentioned to me that in your past experience, you had these Google Docs with definitions and everybody would agree that this is the definition and this is how you calculate it. You were probably the earliest adopter of a semantic layer. This is the space we've been thinking about for the past five years. So, how have you been thinking about it with your customers?
Brian from Hightouch: I would actually rely a lot on my experience at New Relic, where I led marketing and analytics. We were dealing with a publicly traded company, hundreds of millions of dollars of revenue and had a lot of fiduciary responsibility. We really held very deep to do what was right. We didn't have this kind of semantic concept that was properly deployed. Sometimes they were just methodological assumptions we'd made. Other times, we thought we were computing the same metric but, secretly, they weren't the same metric and, actually, quite different. If the table we were referencing was slightly out of sync, it would burn analyst weeks fixing it. Those decisions were multi-hundred million dollar decisions because every 90 days, public companies have to report to the street. We knew if we did this wrong, we were going to be held to account 89 days later.
As I was leaving New Relic, we started to move to a shiny new clean data warehouse in order to create a “zone of blessed metrics” where you are permitted to do any kind of analysis. If you ever need a new “blessed metric”, we would have a collaborative process to create it and to put it in this special zone. This way, we made sure everyone works based on the same official, curated metrics. A lot of it is not tech. A lot of it is collaboration and behavior.
The Future of Data Analytics and The Opportunities with AI
The future of analytics surpasses the current state of traditional analytics and is shifting towards real-time data-driven decisions. Rather than relying solely on analysts to manually make sense of data and make decisions, the goal now is to empower businesses with automated, real-time insights that drive immediate actions. This transition involves leveraging AI and machine learning, to not only analyze historical data but also predict future trends and outcomes. To achieve this, the modern data stack plays a pivotal role by offering a semantic layer that provides context, description, and definition to the data, making it accessible and interpretable for AI applications.
In the BI landscape as well, there are exciting opportunities with AI. AI's potential to predict future trends and outcomes holds great promise for informed decision making. Additionally, the democratization of AI applications is becoming more feasible as technology advances. To use AI's potential effectively, it's crucial to provide data with context, description, and definition. This is where the modern data stack's semantic layer steps in, offering a structured framework that empowers AI to derive meaningful insights from data, enhancing predictive capabilities and making AI-driven decision making a reality.
Allan from Klipfolio: The more data history we have, the richer the data. For example, if a data definition has been consistent for five years, all of a sudden, machines and AI can do a lot more with it. With the ability for Hightouch to bring in more context about each individual record, this must be glucose for an AI machine. What's your take on the richness of data and using AI to further generate insights and guidance for business?
Brian from Hightouch: My particular lens is coming from the place of “value activation”. We already see companies that are leaning in here. I don't want to name the company, but in their customer record table, any single individual customer profile will have upwards of 1,500 predictive analytics traits. This is not science fiction anymore. This is a big Fortune 500 company. They are actively predicting thousands of attributes of everyone they encounter. But, the question is, are those predictions accurate and good? You can't actually apply 1,500 potential personalization opportunities. So you need to figure out ways to slim this down to what is high value in terms of the customer's experience and somewhat intelligible and manageable by the marketing organization that's deploying this. I think that's kind of the next frontier.
I really do think we're just scratching the surface of what's going to be possible here, because rewind two years, those 1,500 traits didn't exist. We're just figuring out how to generate them. And then the next thing is what do you do with them and how?
Allan from Klipfolio: Is the future of BI not really an analytics tool? Is it more of a just-in-time, real-time decision-making support?
Brian from Hightouch: Let’s think about this using an example and in the context of “value activation”. You’re a supermarket, and you've got a grocery app. Potentially, at any given moment, tens of thousands of people interact with that application to figure out their groceries for that week. And, you could have these standardized audiences where you bucket people: people who previously bought popcorn, people who previously bought lettuce or people who previously bought whatever. But, I think the world we are moving towards, is a machine that’s more intelligent than that. It doesn't just look at your past, but kind of predicts your future.
The analyst’s role
Analysts play an important role in business decision making. To super-charge their careers, analysts should think beyond delivering data and start answering questions. Reverse ETL allows data engineers to unlock the value from raw data - streamlining the process of answering business questions and helping the company grow faster.
Allan from Klipfolio: How do we move beyond pure analytics and really influence business outcomes? What's your advice for somebody who really wants to move beyond just using the data to make rudimentary everyday decisions?
Brian from Hightouch: I think I've been privileged to see and actually participate in this for the career development of a number of analysts and people I've worked with. A portion of the value and a portion of the job as an analyst professional or data professional is related to business intelligence and creating wisdom for the company. For example, what's going on? How much money are we making? Et cetera. A career-advancing move is to get into the business, get into the thick of it, and help the CEO, help the CFO, help the director of marketing actually do stuff. No one loses with this strategy, so why don't we all do this?
Allan from Klipfolio: I totally love it. I love this idea of getting into the thick of the business to advance your career because you're actually helping the business grow.
Thank you, Brian - VP Marketing & Growth at Hightouch. It has been a pleasure speaking to you today.
Brought to you by Klipfolio
Klipfolio PowerMetrics is an analytics solution that is built around a semantic layer that abstracts business logic from the raw data in order to create metrics for easy self-serve analysis. To learn more about our platform, visit https://www.klipfolio.com/