Why Shopify App Reviews Miss the Billing Problem

Three independent analyses of 1 and 2-star Shopify app reviews, posted on the Shopify Community forum between August and September this year, landed on the same ordering. Billing complaints came first - roughly 19% to 23% of negative reviews, depending on which analysis you read. "Missing a feature" came last, at about 3%. The reviews are not telling merchants what most merchants think the reviews are telling them.

I keep seeing this in audits. A merchant does their homework, picks a 4.8-star app, installs it, and 90 days later they are on a call with me because a billing line they cannot explain just showed up on their Shopify invoice and the developer has stopped answering email. The review page they consulted before installing is still 4.8 stars. These are two different populations of reviewer, writing at two different moments in the relationship, and the App Store treats them as the same signal.

This is a field-notes post about what I look at now that I did not a year ago.

The reviews you read before installing aren't the reviews you'll leave

One of the sharper findings from rezza.rzk's September analysis of 2,797 Shopify app reviews is a timing split. The median 5-star reviewer wrote their review after 30 days. The median 1-star reviewer had been paying for 90 days before writing anything. Over a quarter of all 5-star reviews land in the first 24 hours after install - before the first invoice even arrives.

That is a design where praise is early and anger is late, and the visible star average leans on the praise. A listing where the complaints have not arrived yet looks the same as a listing where the app is actually solid.

I used to tell merchants to read the first page of reviews. I now tell them to open the "lowest rated" filter first and read ten of those, back to back. The pattern reveals itself quickly: billing, support that stopped answering, something that silently broke. If I cannot find any one-star reviews on an app with 500 or more installs, I check the review dates - usually the one-stars exist, just filtered out of view by sort order and compressed into a small fraction of the volume.

Where the signal is weakest: half the apps have no reviews at all

If a merchant was shopping for a returns and exchanges app last month, they were picking from a catalog of 237 live apps. 126 of them had never received any review at all. Of the 111 apps that had any reviews, 101 showed 4.5 stars or higher, and the median rated app had seven total reviews. Five apps accounted for over half of the 10,345 reviews in the entire category.

That is a signal distribution where "no reviews" is the modal state and "a few reviews, all positive" is the default for everything else. Neither tells you much.

The second thing I check when a merchant shows me a shortlist is whether the listing has moved recently. rezza.rzk also tracked 24,592 live app listings over five weeks in August and September and found that about 3% made any meaningful edit. The median description change was zero characters. A merchant looking at a listing cannot tell from the App Store whether the developer shipped something last week or walked away from the app in 2024.

What I do: open the developer's website, look for a changelog or a recent blog post, check their LinkedIn or whatever account is linked from the support page. If the developer went quiet two years ago and the review page shows the same star rating as a competitor with active commits, that is a tie-breaker against them. There is no native Shopify tool that shows last-update date prominently, and it is the signal most people need.

What I check now, before I install anything

Three habits, in the order I run them.

1. Open the one-star reviews first and filter to recent. Read ten. If more than half mention billing or support non-response, the star average is lying to you.
2. Email the developer a pre-sales question before installing. Not a trick question, a real one. "I am on Shopify Advanced, roughly 2,000 orders a month, considering your app for returns. If I exceed the plan limit mid-month, what happens?" The reply time and quality tell you more about post-install support than any Shopify app reviews will.
3. Note the install date on your calendar at exactly day 85. If the app bills monthly, that is five days before your third invoice lands. Open the invoice when it arrives and check the line items against your actual usage.

The third habit matters because the complaint data says the problem does not show up on day one. It shows up when the second or third charge arrives and the merchant has already stopped paying close attention. By then the switching cost is real: data, custom configuration, maybe theme code embedded in a snippet an app support agent once pasted in and never removed.

If you want a second set of eyes on your app stack before BFCM, the first 15 minutes with a help1 expert is free and that is usually plenty for the triage pass.

The billing mechanic the review page doesn't show

Here is the piece of the pipeline that confuses almost every merchant I speak to. Shopify generates app charges at the start of each 30-day billing cycle, not at the end. The charge is queued, pending, waiting for your next consolidated Shopify invoice. When you uninstall the app, future cycles stop. A charge already queued for the current cycle does not. It will still appear on your next invoice.

This is why so many of the 1-star Shopify app reviews in the data mention being billed after uninstalling. Of the 84 billing-related reviews in Bristan_FARRE's August dataset of 483 reviews, 20 also mentioned uninstalling or cancelling, and nine clearly described a charge that arrived after cancellation. The developer could not void the pending charge on their side either - the Shopify billing API gives developers no cancellation mechanism once a cycle has started. The merchant blames the developer, the developer blames the platform, Shopify Partner Support marks the ticket "escalated" and goes quiet.

A corollary that bites hard: the Shopify billing page does not tell you which app issued a refund. When a developer credits you, it arrives as a generic banner with no app name attached. Developers have been asking for this since early 2025 and the acknowledged workaround is still for the developer to screenshot their Partner Dashboard and send it to the merchant by email. Shopify Staff flagged both the pending-charge problem and the refund-attribution gap as "known limitations" with no ETA.

What I do before any paid app install now: screenshot the install permission screen, note the billing cycle start date, and set a one-line reminder on my calendar. "Did I actually use this app this month? Is the charge what I expected?" It takes 15 seconds. It prevents most of the complaints I see.

The apps that survive every audit anyway

A thread from mid-September asked merchants which apps they ended up cutting after an audit. The pattern in the replies was consistent. The apps that survived were tied to money moving or a legal obligation. Reviews, email, tax, shipping compliance. The ones that got cut could be replaced by a native Shopify setting, a theme section, or a one-time Liquid edit.

One commenter described their test more bluntly than I would have: pause the app for a week and see what actually breaks. If nothing breaks, the app was running on habit, not necessity. If something breaks, you now have exact evidence of what the app was doing for the business.

I started borrowing that test. It is faster than reading reviews, more honest than reading my own invoice line items, and the result tells me which Shopify app reviews I should actually read next - because once I know what the app is doing for the merchant, I know which failure modes to look for in the review history. That is a different search than "what do people think of this app." It is: "where does this app break for people who need the same thing I need?"

For a related angle, I wrote earlier about how the app stack itself can quietly drag conversion in ways the invoice does not show. The billing opacity I am describing here is the sibling problem on the other side of the P&L.

An honest limitation: I have not sat with the raw review text myself. The 23%, 19%, 3% numbers come from three community members (Liber, rezza.rzk, and Bristan_FARRE) who did the manual coding. I trust those numbers because the three independent analyses landed in roughly the same place, and because the pattern matches what I see from the inside of merchant stores. If a fourth analysis landed with a wildly different ordering, I would want to understand why before changing my advice.

The visible star rating is a snapshot of day-30 enthusiasm, read by merchants on day zero, about an app whose billing will not show its hand until day 90.