
A good software selection process moves in one direction: from the problem to written requirements, then to a shortlist, a scored comparison, a hands-on test and a decision you can defend. Most bad purchases skip the first two stages and start at the demo. The fix is to do the thinking before any vendor speaks, and to judge every candidate with the same evidence. This guide walks through eight steps for choosing business software, shows where each one goes wrong, and shows where a ranking built on repeatable AI model evaluations, like the leaderboards on VibeVoted, gives you consistent evidence for the shortlist.
TL;DR
- Start with the problem and how you will measure success, not with a product category or a vendor list.
- Write requirements that are measurable and say what you need, not how the software should do it. Rank them and fix the scoring weights before demos.
- Build a longlist from category rankings, then cut it to three or four using the same criteria for every product.
- Score every vendor on the same scripted scenario, and price the full lifecycle, not just the license.
- Test with your real data before you sign, and plan the rollout before the contract starts.
Table of Contents
- When you actually need a selection process
- Step 1: Define the problem and how you will measure success
- Step 2: Decide who decides, and set a timeline
- Step 3: Write requirements that can be tested
- Step 4: Set a realistic budget
- Step 5: Build a longlist, then a shortlist
- Step 6: Run demos on one script and score them the same way
- Step 7: Price the full lifecycle
- Step 8: Test with real data, check references, then decide
- Where a ranking fits in your process
- For product makers: getting your software evaluated
- Mistakes that derail software selection
- FAQ
- Sources
When you actually need a selection process
Not every tool request deserves a formal process. Before you start one, check whether a tool you already own can do the job through configuration, an integration or a few extra seats. If it cannot, and the gap is real, a structured selection is worth the effort.
Three situations usually justify it:
- A function your team needs is missing, and the workarounds cost more time than they save.
- A current tool creates a compliance or security risk.
- Cost or scale has outgrown the tool you have.
The size of the purchase changes how long each step takes, not whether the step exists. A $15-per-seat project tracker needs a one-afternoon version of this process. A system that holds customer or financial data needs the full version, with sign-off from the people who will own it.
Step 1: Define the problem and how you will measure success
Write one paragraph that says what is going wrong today, who feels it, and what would be different if it were fixed. Then attach two or three numbers: hours spent per week on a manual task, error rate, time to close the books, support tickets per month. These become your success measures and later your test criteria.
Where this goes wrong: teams describe a solution (“we need a CRM”) instead of a problem (“leads go cold because nobody sees follow-up dates”). Describing the solution locks you into a category before you know whether the process, not the tool, is the cause. Talk to the people who use the current system every day. Their frustration is your requirements list in raw form.
If you are still unsure the need is real, the guide on product need analysis helps you test it before you spend anything.
Step 2: Decide who decides, and set a timeline
Name four groups before you look at a single product:
- The owner, who is accountable for the outcome.
- The daily users, who judge usability.
- IT or security, who judge integration and data risk.
- The budget holder, who signs.
A simple responsibility chart (who decides, who advises, who is informed) prevents a late veto from someone who was never asked. Then pick a decision date and work backward from it: when the shortlist is due, when demos happen, when references are checked.
Where this goes wrong: IT is brought in after the shortlist exists. A 2020 academic review of vendor selection by proof of concept lists not involving the IT team early among the common mistakes, alongside unclear requirements, lack of sponsorship and poor change management.
Step 3: Write requirements that can be tested
Requirements are the backbone of the whole process. Everything later, from the shortlist to the scorecard, is measured against them.
The requirements-engineering literature gives clear rules for buying off-the-shelf software. According to IREB’s RE Magazine, each requirement should be clear enough to expose differences between candidates. When you buy rather than build, it should say what you need, not how to build it, and avoid vague words like “fast” or “flexible” that nobody can enforce.
Compare two versions of the same requirement:
| Weak | Testable |
|---|---|
| The tool should be fast | A report on 50,000 rows loads in under 10 seconds |
| Flexible permissions | Managers see only their team’s records; finance sees all |
| Integrates with our stack | Two-way sync with our accounting system, no manual export |
Split the list into must-haves (a vendor that fails one is out) and nice-to-haves (they break ties). Group requirements into functional needs, reporting, security and access control, integrations, and non-functional needs such as performance and uptime.
Turn requirements into weighted criteria
Criteria are the yardstick you apply to every vendor. The CMMI decision-analysis practice says criteria should be traceable to requirements, scenarios or business objectives, ranked on a defined scale, and that the reasons for choosing or rejecting each one should be documented. It also lists risks, business value and total lifecycle cost among the types to consider.
In practice that means a short table: each criterion, a weight (all weights add up to 100%), and a scoring scale such as 0 to 3. Agree the weights before demos. Once a polished demo has impressed the room, nobody can weigh things neutrally anymore.

Step 4: Set a realistic budget
The subscription price is the most visible cost and the easiest to estimate, but it is rarely the biggest surprise. Set the budget as a range for the first three years, not a monthly number. Include implementation, data migration, training, integrations and the time your own staff will spend. Step 7 covers the cost model in detail.
Step 5: Build a longlist, then a shortlist
Now you look at products, and not before. The aim is a longlist of eight to twelve candidates that plausibly meet your must-haves, reduced to three or four for real evaluation.
Sources for a longlist include peers in your industry, your own search, and category rankings. The weakness of most lists is that they mix paid placement, popularity and reviews written under different standards, so products are not judged on the same basis.
This is the step where a leaderboard helps. VibeVoted ranks software products using evaluations from 10 distinct AI models, and every product is assessed on the same seven published criteria: clarity, need, differentiation, evidence, visual design and polish, trust, and sustainability. Our rankings rest on repeatable AI model evaluations against fixed criteria, not on user voting or popularity campaigns, and sponsor payments are exclusively for exposure and do not influence placement. You can check how the scoring works on the rules and methodology page.
Here is how to use it for a shortlist:
- Open the category that matches your problem and read the side-by-side leaderboard.
- Look at the criteria that match your priorities. If you handle sensitive data, trust and evidence matter most to you. If the tool is for a wide group of non-technical staff, clarity and visual design and polish matter more.
- Read the model-generated feedback for each entry to see strengths and weaknesses from several AI perspectives instead of one.
- Check the weekly and monthly rankings to see which products keep performing over time.
- Carry the best candidates into your own scorecard from Step 3. The ranking tells you where to look first. Your requirements decide who stays.
Readers do not pay to use the rankings. They exist so a buyer can get a comparable, criteria-based view of a category in minutes instead of days.
If quality is your main worry at this stage, the guide on how to assess software quality pairs well with the leaderboard.
Cut the list with a short questionnaire
Send the longlist a short request for information built from your highest-weighted requirements. Use closed questions (“Does the product support single sign-on? Yes, no, or roadmap with a date”). Reject any vendor that fails a must-have, and keep the answers in writing so you can compare them side by side.
Step 6: Run demos on one script and score them the same way
If each vendor runs its own best-case demo, you are comparing sales skills, not software. Write two or three scenarios from your real work, send them to every vendor in advance, and ask each to show those scenarios in your order. Use the same attendees, the same questions and the same scorecard every time.
Then:
- Have each evaluator score independently, with a written reason for each score.
- Multiply each score by its criterion weight. A score of 4 out of 5 on a criterion weighted 20% gives 80 points on a 500-point scale, so the arithmetic is the same for every vendor.
- Mark missing information as unresolved, not as poor fit, and chase it.
- Compare totals only after all demos are done, then discuss gaps between evaluators.
Where this goes wrong: a tick-box yes/no sheet. Two products can both “support reporting” while one does it in three clicks and the other needs a consultant. Scoring relative capability against weighted requirements is what separates them.
Step 7: Price the full lifecycle
A UK Cabinet Office-backed study at the London School of Economics defines total cost of ownership as all the costs of identifying, acquiring, installing and operating software, plus exit and migration costs. It identifies 14 cost drivers across five stages: selection, acquisition, integration, use and retirement. The software itself was the largest and easiest factor to estimate, followed by maintenance and upgrades, then contracted support. The study also found that most organizations surveyed did not use total cost of ownership as a formal decision tool. Another reason to do it: you will be ahead of most buyers.
Build one table for every finalist, covering years one to three:
- License or subscription, including per-seat tiers, modules and multi-year discounts.
- Implementation: vendor hours, your staff’s hours, fixed price or time and materials.
- Data migration and integrations.
- Training for the number of people who will really use it.
- Support and upgrades.
- Exit costs, such as export formats and contract notice periods.
A 2023 multi-criteria evaluation paper adds a useful split: direct expenses include purchase, training, third-party services and IT staff time, while indirect costs include user self-support and downtime. It also says return on investment is needed next to cost to judge the economic effect. Tie the result back to the success measures from Step 1: if the tool must save 20 hours a month, what is that worth?
Give every vendor the same pricing template. Otherwise a “cheaper” proposal may simply leave out implementation or training.
Step 8: Test with real data, check references, then decide
A demo shows what the software can do. A proof of concept or trial shows what it does in your environment. The FH Wedel review notes that the advantage of a proof of concept is proving the software meets your requirements in your own system landscape, and that even an unsuccessful one surfaces errors and inconsistent definitions early. It also lets you stop at low cost before you commit to a license.
For a trial to mean something:
- Use real, messy data, including the odd cases, not the vendor’s sample set.
- Involve IT early to test integration, access control and security.
- Let daily users run the scenarios from Step 6 and rate usability.
- Set a fixed time limit, because trials that drag on lose focus.
- Call at least two reference customers of similar size, and ask what went wrong during rollout.
Then choose. Add up the weighted scores, compare with the three-year cost, and document why the winner won. That record protects you when someone asks six months later. Negotiate on term length, price caps on renewal, support levels and exit rights before you sign.
Finally, plan the rollout before the contract starts: who migrates data, who trains whom, what the first 30 days look like, and when you will check the success measures from Step 1.
Where a ranking fits in your process
A ranking is evidence for one stage, the shortlist, and it works best when you treat it that way. The strength of a consistent evaluation is that every product in a category has been looked at through the same lens, so the differences you see come from the products and not from how loudly each was promoted.
For a buyer, that saves the slowest part of the process: working out who even belongs on the list. For a product manager, it gives a neutral starting point to defend in a meeting. For a developer choosing tooling, the model-generated feedback highlights strengths and weaknesses to ask about in the demo.
For product makers: getting your software evaluated
If you build software, the same process runs in reverse: buyers are searching leaderboards for exactly the problem you solve. On VibeVoted, makers sponsor a spot to enter a product into the ranking and receive model-generated feedback on each entry. One voting session costs 25 USD, one-off per credit. Sponsor payments are for exposure only and do not influence placement.

You can see how entries are listed on the VibeVoted homepage and enter a product through the sponsor page. If you are earlier in the journey, the guides on validating a SaaS idea and promoting a SaaS product cover the work before and after listing.
Mistakes that derail software selection
The academic review of proof-of-concept selection lists the mistakes that recur: no real data, IT involved too late, unclear requirements and deliverables, no sponsor, poor change management, unclear cost estimates and timeframes that run too long. Add three that show up in nearly every project:
- Starting with demos. The most important work happens before the first one.
- Changing the weights after seeing scores. If a result feels wrong, find out which requirement you under-weighted and document the change openly.
- Forgetting adoption. The best tool on paper fails if daily users avoid it. Include usability and training in the criteria.
FAQ
What are the steps in the software selection process?
Define the problem and success measures, agree who decides, write testable requirements, set a budget, build a longlist and shortlist, run scripted demos with one scorecard, price the full lifecycle, then test with real data, check references and decide. Plan the rollout before you sign.
How long should choosing business software take?
It depends on the stakes. A low-risk team tool can be settled in days. A system holding financial or customer data usually needs several weeks to a few months. Set a decision date at the start and work backward from it.
How many vendors should I evaluate?
Start with a longlist of eight to twelve, then cut to three or four for demos and trials. Fewer than three leaves you without a real comparison. More than four makes every evaluator’s job heavier without improving the decision.
How do I compare software fairly?
Use the same requirements, the same weights, the same demo script and the same pricing template for every vendor. Score independently before discussing. For the shortlist, a category leaderboard that judges every product on the same published criteria gives you a consistent starting point.
Sources
- IT Requirements when Buying, not Making, IREB RE Magazine
- DAR.SP 1.2 Establish Evaluation Criteria, CMMI
- Total cost of ownership of open source software, LSE Research Online
- Multi-criteria evaluation of IT investments, cal-tek proceedings
- Software Vendor Selection Using Proof of Concepts, FH Wedel