Kaushik Kannan | Jun 28, 2024

Kaushik Kannan (00:00) I actually was a member of our e -commerce marketing team and was able to launch an A -B test that generated just under $2 million Ari Block (00:09) talk about the hypothesis experiment that was driving this whole thing. So kind of break it down for us. Kaushik Kannan (00:14) Yeah, absolutely. So the page that I was working on was an informational page for balance transfer cards, right? So people moving from one credit card to another credit card product and trying to understand how they could transfer a balance from one credit card to the other. The problem that we were trying to go after was this page was very informational, but it was receiving quite a healthy amount of traffic which was actually pretty high for one of these what we called out of flow pages that weren't part of the traditional funnel. And so we identified an opportunity here and the hypothesis was if we could add conversion focused CTA, so buttons, encouraging users to get pre -approved for credit card products or actually apply for those credit card products and get approved, then we could drive more conversion, right? So taking this page from an informational page to something that was more conversion focused. would lead to more conversion. That was the hypothesis. So work with some business analysts and whatnot to formulate that, pitch it to leadership, and move forward. Ari Block (01:15) That's super interesting. What happened there? How did they receive it? Kaushik Kannan (01:19) they also understood that there was an opportunity here to port over some of the conversion -focused components from other pages. that had actually worked in generating conversion, right? So that gave them more credibility and that gave, that also gave them more confidence to move forward You know, I think that really helped them overcome any initial fears. But when it came to our collaboration with cross -functional partners, I think there was some hesitation initially What component are we using? What sort of dependencies would there be on other pages and things like that? And I think the way that we addressed that was really focusing on this opportunity in particular and saying, this page is something that could use this update right now. And based on what we learned here, we can move forward, right? And fears. Ari Block (02:05) I love that. You're just glossing over an incredibly important, let's say, product management tactic, which is narrowing the scope, making the scope very clear and focused, and kind of focusing the discussion around rejects, what are the concerns, what are the opportunities. That's beautiful. I mean, that's just, I think that's one of the fine parts of the art of product management. I really appreciate that. Thank you. So, okay, so you had to get steak. Kaushik Kannan (02:27) Yeah. definitely. Ari Block (02:30) stakeholder buy -in, right? That's what we do. You had to form your assumptions and understand what they were. You formed a hypothesis and this was a nice one. This was clear cut. It had conversion. What was the conversion number and percentage that moved from where to where? Kaushik Kannan (02:45) the lift that we generated in present value per visitor was around $1 .2 million, And that was our key metric that we're trying to move, which was like, how much value do we think we can capture over the lifetime of some potential prospective customers account that is now converting because they came to this page, saw the CTA and took some action. Right. And so that was sort of the needle that we moved was adding these conversion focused things. And we saw that the lift was that amount of money. Ari Block (03:15) Amazing. So tell me a little bit more about the tooling. Kaushik Kannan (03:19) Yeah, absolutely. So the tool that we use was called Adobe Target. It's part of the Adobe Experience Cloud suite of products. And it was a product that existed prior to my time at Capital One, so not something new that we had to buy into. And one thing that I really liked about it was it was really simple to start building these tests. There was not a lot of like back end configuration work or even front end complexity for that matter. Like people like myself were the ones that were creating this and we were not developers, right? We're not writing code or anything like that. Ari Block (03:50) Wait, hold on. You created the A -B testing yourself without developers? Okay, so this should be the biggest sell, I think, to one of the biggest sells to product managers, except that incredible uplift in revenue is that really you're empowering the product managers to run tests based on hypotheses and assumptions. Kaushik Kannan (03:57) did. Ari Block (04:11) without waiting three or four weeks for engineering or less or more, depending on your organization. Kaushik Kannan (04:12) Exactly. Ari Block (04:16) So that's amazing. Let's give a little bit of free press here for Capital One, Kaushik Kannan (04:21) Capital One, Capital One, yeah. Ari Block (04:23) Yes, wonderful. Well, way to go, Capital One. Okay. So that's amazing. But one of the big things we do is we get buy -in from stakeholders. Tell me about one of these times where that wasn't an easy task. Kaushik Kannan (04:36) we were working on introducing an FAQ section to one of our credit card product pages, which was for our student credit cards. So we have various card suites for specifically students that are looking to get started with credit. Normally, I haven't had credit in the past. And we wanted to add an FAQ section to that page because we anticipated that there might be some common questions. And that was something that we had on all of our other Card Suite pages that we thought could be a helpful addition. So the buy -in challenge here was with the students team. So we actually had a part of our business that was working on these student card products to increase adoption and do marketing, stuff like that. And I think there was a challenge there because they were very particular about the types of copy that they wanted to use for these FAQ sections. So obviously two different perspectives. And I think the way that I slowly built that trust was by bringing both teams together. I think in product circles, there's often people work in silos, Where one team is working by themselves and the other team is working by themselves. And then they put something together, but they don't talk. And only at the end, they realize they've been going in different directions the whole time and wasted time. to prevent that waste of time, I developed a cadence for actually connecting the teams and resolving any issues or dependencies and showing our work. Ari Block (05:50) Nice. Okay, so that's such a wonderful story, I appreciate that. And there's a keen insight here and that's that people matter, meeting face -to -face matters, listening matters. Kaushik Kannan (05:53) OK, cool. Ari Block (06:03) It's and I think I feel like it's something that product managers learn very, very early on and you executed beautifully. You know, when I was early in my career, I was talking to CEO and I'm like, this project is not working, you know, and I was all over WebEx, right? That was what we did back in the day. And he said, okay, you're going to Estonia, you're going to meet the team, you're going to buy them beer, you're going to have dinner with them, you're not going to work. That blew a fuse. I was like, what do you mean I'm not going to work? He said, just go break bread. And as a young product manager, I was like, that's not... I was ready to come back and tell him, you were wrong, I was right. I had an ego back then. I come back from Estonia, we worked for 30 minutes, like it was nothing. And then three weeks later, the project's done. I had been at it for three months. Three weeks, project's done both sides. So I come into the CEO's office and I'm like... How did I miss this? Why did this go so badly? And then obviously he explained to me about egoistic bias, where I basically was judging them based on my culture, when really I was being disrespectful in the fact that I didn't break bread. So like, who's this guy? What does he want from us? But by showing them that respect, I created a cohesive team. So I mean, that's, it's just beautiful. That's what I learned early in the career. And clearly you have your story where you learned that lesson. And I think it's such a huge tool for product managers. So I really, really appreciate that ability to build teams. That's wonderful. I want to, I want to pivot a little bit. One of the big things product managers do is about prioritizing. Kaushik Kannan (07:32) Yeah, thank you. the way that planning process worked is I would aggregate together particular feedback requests that have come through from customers through our Pendo feedback tool. So Pendo is one of these like data analytics and customer feedback platforms where customers can express feedback requests. So it combed through that, groomed that backlog, understand if there were any pressing things that we could address for customers, and then write out tickets for that and plan them out. I'd also comb through our existing backlog. So this is a way that we plan in the short term to fix anything that was... waiting in the backlog for a long time in terms of bugs or evergreen stories or things like that that we hadn't gotten to in previous sprints. And then I'd also think about carryover. So I think about what developers are working on in the current sprint that based on the updates that I'd gotten weren't things that we could complete in the current sprint and needed to be moved over. So that was a way that I'd assess how I could capacity plan for the short term. And then Combining that with the long term, the way we would look at that was, if there were any big features that we needed to build, just really scoping those out by doing a tech spec as well as a product requirements document to understand what does this epic look like. So we take the requirements document, break it into an epic and understand what delivery of that epic looked like. How did that strategy look? And then figure that out on our roadmap, right? Like visualize that on a roadmap so that our stakeholders could see that. So that was short -term and long -term planning there. And then real fast with Capital One, before I give it back to you, we had another intake process where we'd have an intake meeting once a week, business partners would come and discuss requests with us, so we would take in those requests. We'd also have, we used Google Sheets, we were kind of old school like that, so we would use a big Google Sheet, which was our backlog of testing items and understand what we could pull into the next sort of sprint. Ari Block (09:02) Wonderful. Kaushik Kannan (09:27) PMs could run these A -B tests. So it really didn't affect capacity, but it did affect like business analyst capacity and stuff like that, which wasn't necessarily developers. Ari Block (09:35) That's amazing. You know, some of our audience are CEOs and VPs of marketing. And, you know, we've been very technical, so to speak, at the product, let's say, management leadership level. I want to take a second to break it down. So there is a process, right? We call that grooming of the backlog. And really what that is, it's taking those small features and picking the most important, the most impactful ones, the ones that are going to have the biggest pain and affect the most users, right? not the one that one user has and nobody will care because we're not a consulting shop, right? We're a product shop. It's got to meet 80 % of the user's needs. But then there's also a longer term process, right? And that's the themes or the epics. Tell us for our non -product manager listeners, tell us a little bit about what the difference between an epic is and the backlog grooming and how that ties up to, let's say, the strategic objective. of the organization? How do those three things all tie together? Because what I find is a lot of people coming into new organizations, they don't necessarily have this wonderful product culture that you've been described. Kaushik Kannan (10:42) Yeah, absolutely. So I think with epics, the way I like to think about it is I think we should first define a story because I think of an epic as a collection of stories, right? So when we say user stories, what I really mean by that is a document in the highest possible level, right? So for looking high level, the document usually prepared in some project management software like JIRA or Trello or some software like that, where we are trying to communicate as product managers what we would like to get built in terms of the current state, the future state, things like that, right? So this can include like a screenshot of the existing state saying on this particular screen, we would like to see a button that is blue that says, see if I'm pre -qualified for X, Y, and Z credit card product, right? It can be something as simple as that. And then the desired state is a mock -up showing that button added to the screen. where we are communicating design intent for the developer to say the button should be placed here, it should be X by Y in terms of its dimensions, it should be this color, right, defining it in terms of the hex value, stuff like that. So we're getting very granular about what we wanna see. Then when we go to the Epic level, that's really a collection of these stories, right? So this could be as part of the Epic could be something like we wanna redesign the onboarding flow for This app right which is a much bigger initiative that can't be covered in a single story You could cover it, but it would be a very long story. It would be very hard to understand So epic really organizes these stories for us into something that layers up into a roadmap item or what we call like a big rock, right? That's what we use that logic gate was really these bigger product initiatives that we communicate across the company and say this is what we're working on right because nobody really cares We're working on a button, but they do care if we're working on a redesign of the onboarding flow Ari Block (12:37) right. Kaushik Kannan (12:37) So Ari Block (12:37) just such an important point because the rocks and the pebbles or the sand, really that's what connects you to the OKRs, right? The objectives and key results of the strategic infrastructure of the organization. And I think that's something that, you know, non -product leaders, they don't always fit into that language, that there's a strategic objective for the organization that connects to product objectives. And then there's a hypothesis behind that. That hypothesis is then supported by a product experiment. So that path that you've just described, how that goes kind of up, down and back up is so incredibly important. I find that it's not necessarily common knowledge for all let me say thank you so much and we will definitely talk again.