Rebuilding the Quiz editor used by thousands of Shopify brands as AI-native

Role: Product Manager · Octane AI Scope: End-to-end redesign of the Quiz editor used by thousands of Shopify brands, rebuilt to be AI-native Team: Worked with the CTO on the AI runtime, plus design and customer-facing teams

TL;DR

• The Quiz editor at Octane AI is a mature product thousands of Shopify brands rely on. It worked, but it had aged into something new merchants struggled to use.
• I recorded my own first-use friction, then grounded it with an NPS survey: 145 merchants, NPS of +14, with usability the loudest complaint. Self-serve Basic merchants scored -15, against +33 for managed Enterprise.
• Approval came with a condition that raised the ambition: make it AI-native, so anything you can do in the editor you can also do by chatting with it.
• I led the redesign end to end. This case study focuses on that work: the before, the after, and how every surface was rebuilt.

Context

Octane AI is an AI-powered e-commerce platform built for Shopify. The Quiz editor is one of its core surfaces: it's how merchants build the quizzes that turn first-time shoppers into segmented customer profiles, used downstream for personalized product recommendations, email flows, and conversational follow-ups.

Thousands of Shopify brands use it daily. By the time I joined as PM, it was a mature product that had also accumulated years of friction. The hard part of redesigning something like that isn't the visuals. It's earning the right to touch a tool that's already generating revenue without breaking what works.

The problem

My first week was onboarding. In my second, when I finally opened the editor to build my own first quiz, I recorded the screen. I already suspected that this first-time frustration was a perishable asset: the clearest signal I'd ever get, and one I'd lose the moment I got used to the tool.

My first attempt building a quiz in the old editor, recorded during my second week.

In the clip I set out to do the simplest possible thing: collect the shopper's name with a single text field. It took minutes of trial and error. I couldn't work out how to delete a question I didn't want, I kept landing on screens I wasn't looking for, and I got tangled in the difference between a question's "internal name," its "title," and the page name. You can hear me reason through it in real time: "where do I remove this page?", "no, that's not what I'm looking for," and eventually just "okay, that's confusing." The problem wasn't a lack of power. It was that a motivated, technical user couldn't form a simple intent and carry it out.

But one person's bad afternoon isn't evidence. So before proposing anything, I ran an NPS survey on the current editor to ground the decision in data instead of my first impression. 145 merchants responded.

The result was an NPS of +14. Just under half were promoters (48%), but a third were detractors (34%), and the comments showed why. The scores were bimodal: 64 merchants gave a perfect 10, while a heavy tail sat at 5s and 6s. That's the fingerprint of a tool people want to love but that fights them on the way in.

The loudest theme, by a wide margin, was usability: "needs to be more user friendly," "too difficult to use," "takes a learning curve." The second was design rigidity: "too rigid," "design capabilities are a problem," "customisation rigidness." That second one had a structural cause. Octane AI is a ten-year-old company that never staffed design as its own function, and the development team turned over repeatedly in that time. Features shipped close to however the original mockups arrived, without a design owner to maintain coherence across them. Over a decade, that compounds: the surface accretes into something hard to bend to what each individual merchant needs.

The business impact was real, but easy to miss in the top-line number. Our internal sales platform reported churn at around 5%, which is suspiciously clean for SaaS and didn't match what we were watching happen: the editor's learning curve was pushing new merchants to give up before they reached value. Splitting the same NPS survey by plan made the gap concrete. The self-serve Basic tier ($50–$200/mo) scored an NPS of -15, while the managed tiers (Enterprise, Unlimited), where every customer gets an account manager who walks them through onboarding, scored +33. The blended +14 hid a 48-point spread. You can see the mechanism in the comments: several of the warmest Enterprise responses thanked their account manager by name, not the editor. Basic merchants had no such safety net, and that's exactly where the churn was. The editor was failing the customers who had to face it alone.

And the editor isn't a side feature. It's the entry point: if a merchant can't build a quiz, nothing downstream (recommendations, email capture, personalization) ever fires.

Making the case

A pattern this visible doesn't usually need a long investigation. The harder problem was credibility: a PM in month one proposing to rebuild a revenue-generating product is a fast way to get ignored. So I led with the evidence instead of opinion: the recording first, then the survey.

Approval didn't come back as a simple yes. My manager raised the bar in exchange: he'd greenlight the redesign if it was AI-native from the ground up, with every editor action also reachable through a chat panel. Describe what you want ("add a blue background to the second page") and the editor makes the change.

That condition became the project's defining bet. It's also what got the CEO behind it: not "fix the editor," but "ship an editor no competitor on Shopify has."

The redesign

The brief I set was simple to say and hard to do: make the editor something a first-time merchant can actually use, without taking power away from the merchants who already relied on it. The work touched every surface. Here is the before, the after, and how each piece was rebuilt.

The dashboard, before and after

Before: the old dashboard.

After: the redesigned dashboard.

The editor, before and after

Before: the old Quiz editor.

After: the redesigned editor.

The canvas

The old editor buried structure in nested panels. The redesign puts it on the canvas: direct editing, the page laid out the way it will actually appear, and far fewer clicks between intent and result.

Editing on the redesigned canvas.

Components

A real component model replaced one-off screens: reusable blocks a merchant can drop in, rearrange, and restyle, with the same pieces powering both the editor and what shoppers see.

Building with the component model.

Live preview

You see the shopper's experience as you build it, instead of guessing and publishing to check. The gap between "edit" and "what the customer gets" closes to nothing.

Live preview of the shopper experience while editing.

AI-native, not AI-bolted-on

The reason a chat panel can do anything the UI can isn't a feature bolted onto the side. It's architectural: every change in the editor flows through a single command layer, so the AI assistant emits the same commands a human would when they click and drag. Nothing the AI does is a special path; it's the editor's own vocabulary, spoken by a model.

My role on this part was product, not engineering: defining what "AI-native" had to mean for a merchant, and making sure the editor and the assistant never drifted into two products that behave differently. The result is an editor that's AI-native by construction rather than by integration, which is rare on Shopify today.

The AI assistant editing by chat: "add a blue background to the second page."

Where it landed

I left Octane AI at the end of June 2026, before the redesign reached internal alpha testing. That means I can't report post-launch numbers — no adoption curve, no NPS delta — and I won't invent them. An honest case study ends where the author's knowledge ends.

What I can report is what was true when I left: the full front-end of the new editor — canvas, component model, live preview, and the AI assistant — was built and in integration with the platform's backend, and the evidence-first playbook that unlocked the project (record the friction, survey the base, split the numbers by plan) had become the template for how the redesign was argued for and scoped.

How I work

Strip away the specifics and this is how I operate on any product.

Treat first-use friction as a perishable asset. You get one chance to see a product with fresh eyes, and it expires the moment you get good at it. I recorded my own first attempt at building a quiz in week two, fumbling and all, because that confusion was the clearest signal I'd ever have and I was about to lose it for good.

Lead with evidence, not authority. A PM in month one has no standing to rebuild a product that's already making money. So I didn't argue from opinion. I let the recording and a 145-merchant survey make the case, and grounded every claim in something I could point to rather than something I felt.

Lower the floor without lowering the ceiling. The way you ruin a mature tool is by trading one set of frustrated users for another. The whole brief was to make the editor usable by a first-timer without taking anything away from the merchants who already depended on it. That tension, not the visuals, was the real design problem.

Prototype it myself. I don't hand off a wireframe and wait for someone to build it. I made the first working version of the redesign by hand: I started in Google AI Studio while I got comfortable prototyping by vibe coding, then moved to Claude Code entirely once the workflow clicked. Staying that close to the code lets me pressure-test an idea before anyone spends real engineering time on it.