Tallyby Outpost

Theme

Start for free

Developer tools

For testing tools: which framework guides and CI integrations bring teams that pay for runs.

Testing tools are picked by QA engineers and developers who wire them into CI, run them on a trial and pay for test minutes, parallel runs or seats once the suite depends on them. Tally ties each plan and usage invoice to the visits behind it, including the framework guide that got the first test running.

Start for free

14 days free, no card needed.

01How they sell

  • UsageTest minutes, parallel sessions or virtual users for load tests, billed monthly with overage.
  • SubscriptionTeam plans by seats or parallelism, monthly or annual.
  • FreemiumA free tier for open-source projects or small suites, with limits on runs or history.

02Payments it reads

  • StripeUsage-based and team plans through Stripe; overage invoices, renewals and refunds are read.
  • PaddleTesting tools selling to companies worldwide use Paddle as merchant of record; transactions and adjustments are read.
  • PolarSmaller testing tools for developers bill through Polar; orders and subscription cycles are read.

What Tally can't read

  • CI minutes your customers pay their CI provider aren't read.
  • Plans bought through GitHub Marketplace are billed by GitHub and aren't read.
  • Test results, screenshots and recordings in the product are never read.

03Where buyers come from

  • Framework guidesSearches for how to run Playwright or Cypress tests in parallel, or how to add visual tests, land on your guides.
  • CI integration pagesEngineers wiring a test run into GitHub Actions or another CI service find your integration page.
  • Conference talksTesting talks send engineers who look the tool up afterwards; Readable Direct counts those who type it in.
  • QA communitiesForums and Slack groups for testers recommend tools by name.
  • Flaky test searchesSearches about flaky tests bring engineers ready to pay to stop the noise.

04What Tally answers

  1. Do teams that run a first test in CI go on to pay?

    Send tally('first_ci_run') from the dashboard when results from CI first arrive; the goals view counts those visitors, how many paid and their revenue.

    Goals and funnels

  2. Which framework guide sells, Playwright or Cypress?

    Guides appear in the page breakdown with their visitors and the revenue credited to buyers who landed on them, so each framework's guide is judged by sales.

    Sources with revenue

  3. How long is the trial before a team pays?

    Path to purchase gives the median time and visits from first visit to first payment, with a histogram of the spread; a trial that waits on a merged pull request before anyone pays stretches it.

    Path to purchase

  4. Which visits came before the annual contract from that test lead?

    The contract's receipt lists the visits behind it, with the credited one marked and how the payment was linked, so the guide or talk that started it is named in the evidence.

    Receipts

  5. Where do trial users get stuck setting up?

    Session replay, once turned on and with inputs masked, plays back setup sessions in the dashboard, so you can watch where engineers stall between installing and seeing a result.

    Session replay

05Numbers worth watching

Goal conversion
The share of visitors whose first CI run arrives, the step after which a suite starts to depend on you.
Median time to buy
The median time from first visit to first payment, the length of a team's trial in practice.
Revenue per visitor
Revenue per visitor by guide and integration page, for choosing which framework to write about next.
Visits before buying
Visits before paying, which counts the docs trips needed to get a suite running.

06An example

Example

Path to purchase

Last 30 days · 105 of 124 new sales, the ones linked to a visitor

Median time to buy

3days

from their first visit

Median visits

3visits

the one they paid in included

Bought on the first visit

12%

13 of 105

Pages before buying

11

median pages viewed

Where buyers came from, and who closed the sale

105 new sales: where each buyer first came from, and the source credited with the sale.

50% of new sales were credited to the source that first brought the buyer in. Hover or focus a flow for its numbers.

  1. Googlecredited toGoogle26
  2. x.comcredited tox.com10
  3. Other (3 sources)credited toOther (3 sources)10
  4. Directcredited toOther (3 sources)9
  5. ChatGPTcredited toChatGPT9
  6. Other (3 sources)credited toNewsletter9
  7. Redditcredited toReddit8
  8. Googlecredited toNewsletter6

6 smaller flows in the table below.

Show the flows as a table
105 new sales: where each buyer first came from, and the source credited with the sale.
Brought them inCredited with the saleNew salesShare
GoogleGoogle2625%
x.comx.com1010%
Other (3 sources)Other (3 sources)1010%
DirectOther (3 sources)99%
ChatGPTChatGPT99%
Other (3 sources)Newsletter99%
RedditReddit88%
GoogleNewsletter66%
DirectGoogle55%
x.comGoogle55%
RedditGoogle33%
DirectReddit22%
ChatGPTGoogle22%
Other (3 sources)ChatGPT1<1%

Time from first visit to purchase

  1. 13Same visit: 13 sales (12%), bought in their first visit
  2. 13Same day: 13 sales (12%), a later visit, under 24 hours after the first
  3. 301–3 days: 30 sales (29%), 1 to 3 days after the first visit
  4. 194–7 days: 19 sales (18%), 4 to 7 days after the first visit
  5. 151–2 weeks: 15 sales (14%), 8 to 14 days after the first visit
  6. 102–4 weeks: 10 sales (10%), 15 to 28 days after the first visit
  7. 51–3 months: 5 sales (5%), 29 to 90 days after the first visit
  8. 03+ months: 0 sales (0%), 91 days or more after the first visit

Pages buyers saw before paying

Share of the 105 buyers who viewed each page before paying

  1. /pricing96%, 101 of 105 buyers
  2. /71%, 75 of 105 buyers
  3. /signup63%, 67 of 105 buyers
  4. /docs38%, 40 of 105 buyers
  5. /blog/stripe-fees26%, 28 of 105 buyers
  6. /compare21%, 23 of 105 buyers
  7. /changelog8%, 9 of 105 buyers
  8. /blog/launch-week5%, 6 of 105 buyers

Only new sales count here. Renewals and upgrades belong to the visit that won the subscription, so counting them would count that journey again.

Fig. 1Path to purchase on the example site. For testing tools, look at how long trials run before payment; the histogram shows whether teams decide in a week or after a quarter.

07Setting up

  1. Next.jsIf docs and dashboard are one Next.js app, add the tag to the root layout so guide visits and in-app goals belong to one visitor.
  2. StripePass tally_visitor_id in the Checkout Session's metadata and subscription_data when a team adds a card, so overage invoices follow.
  3. Call tally('identify', { email }) when a user joins a team, so an invoice paid later by the team lead with that email links.

08Questions

  • Do test runs count as visits?

    No. Runs in CI and on your infrastructure aren't visits; only pages that carry the tag are counted.

  • Are monthly overages credited to the source that won the team?

    Yes. They bill on the team's subscription, so each takes the link of its first payment.

  • A QA lead asks for an annual invoice. Is that read?

    When it's a Stripe or Paddle invoice, yes, linked by email if the lead was identified on your site; a wire transfer outside them isn't read.

More in developer toolsEvery business