Methodology for Quadrants:

Given the prevalence of quadrant-based market ratings throughout IT,  many will consume our research without a deeper investigation into our methodology at Cloud Ratings.   

Cloud Ratings Quadrant process combines private vendor responses to our Request For Information (RFI) with publicly visible measures of market adoption and customer satisfaction.

All vendors are presented with the opportunity to complete the RFI. 

The Cloud Ratings RFI focuses on measures of:

  • Customer Satisfaction: Greatest weight is given to internal Net Promoter Score (NPS) data. Why? Certain software vendors are skilled at driving (and often incentivizing) happy customers to submit reviews to popular software directories. This can lead to situations where publicly observable vendor reviews are more favorable than the “silent majority” of customers that do not submit to software review platforms. As such, Cloud Ratings’ use of private NPS data provides a more holistic view of customer experiences. To measure adoption and implementation risk, significant consideration is also given to vendor responses regarding severely negative customer outcomes and retention rates.
  • Market Adoption: The RFI covers a range of traction metrics, including but not limited to revenue, revenue growth, and the current number of customers. Why? Vendors can appear to have an artificially high market share through aggressive marketing spend, public relations, and skillful solicitation of reviews to software directories. Accordingly, Cloud Ratings’ use of RFI-derived information helps provide software buyers with a more realistic view of market adoption.       

This graphic – which compares the number of customer ratings provided via vendor NPS data against that vendor’s aggregated reviews on leading software directories – highlights the enhanced market coverage provided by Cloud Ratings RFI:

G2 Data Partnership: Starting in 2024, Cloud Ratings partnered with G2 to add enhanced data for ROI, implementation timelines, and customer segment mix. However, Cloud Ratings methodology for calculating “publicly visible measures of market adoption and customer satisfaction” is unchanged by this G2 data partnership. Said differently, G2 reviews receive no additional weight in the Cloud Ratings quadrant mapping and ranking process. For clarity, G2 has zero editorial input over Cloud Ratings quadrants. Furthermore, G2 benefits from all provisions set forth in the “Cloud Ratings Disclaimer” section. 

Definitions:

Shaded Areas of Category Chart: Each category of software has a unique likelihood of generating business value and positive experiences for customers. Accordingly, Cloud Ratings customizes the shaded surface area for each category as a visual representation of the zones of positive customer outcomes. A category with higher risk or lower satisfaction ratings (like ERP or HR software) will have smaller Zones of Excellence and Zones of Solutions than a safer, higher satisfaction category.  For Direct Mail Automation Software, a normalized customer rating score of 4.60 (on a 5.00 scale) was used in setting the Zones’ surface area.  

Zone of Excellence (Light Green): This represents a category’s strongest vendors as reflected by their combination of market share and high customer satisfaction scores.   

Zone of Solutions (Light Yellow): Encompasses vendors with both meaningful marketplace adoption and sufficiently high absolute customer satisfaction ratings.

ROI via G2: G2 asks product reviewers, “How long did it take to go live?” For the ROI gauge featured on each quadrant summary, Cloud Ratings takes the overall category ROI provided in the most recently available G2 Grid report and then compares that ROI against a universe of G2 category ROIs. For Vendor Profiles within this report, the ROI (as measured in payback period months) is inputted from the most recent G2 Grid report.  

Implementation via G2: G2 asks product reviewers, “What is your organization’s estimated ROI on {reviewed_product} (payback period in months)?” Cloud Ratings calculates the median and average category time to live provided in the most recently available G2 Grid report and then compares that implementation time data against a universe of G2 category timelines. 

Customer Size via G2: G2 classifies reviewer organizations into three groups: “Small Business: 50 or fewer employees”, “Mid-Market: 51-1,000 employees”, and “Enterprise: more than 1,000 employees.” For the Customer Size row chart on each quadrant summary, the category’s average customer segment data is inputted from the most recent G2 Grid report. For Vendor Profiles within this report, the Customer Size Mix also reflects the most recent G2 Grid report. 

Momentum: Derived from sources like RFI-sourced vendor growth metrics and buyer search interest, the Momentum indicator compares a category’s growth rate with overall information technology (IT) spending growth. 

Risk: Influenced by RFI-sourced vendor NPS and adverse outcomes data, the Risk indicator compares the risk profile of a software category to software applications generally. The distribution of strongly negative public user reviews also factors into the Risk indicator level.  

Notable Vendor: This distinction highlights other vendors identified in the research process as worthy of buyer consideration. Common reasons for Notable Vendors to not be featured in the Category Chart include differing levels of product focus (a secondary feature within a broader suite) or comparatively lower market adoption metrics, especially for earlier-stage companies.      

Cloud Ratings Disclaimer:

Cloud Ratings research analysis and publications represent opinions – expressed at a specific moment in time – and should not be viewed as statements of fact. 

Cloud Ratings is not responsible for any incorrect information supplied by vendors, customers of vendors, or derived from publicly accessible information. 

Cloud Ratings assumes no liability for damages resulting from the application or usage of information, content, or research. 

Cloud Ratings disclaims all warranties as to the commercial success or outcome of any resulting business activities, vendor selections, or investment decisions from the information provided. 

Cloud Ratings does not endorse any solution or service named in its research.

All product names, logos, and brands are the property of their respective owners. All company, product, and service names used in this report and website are for identification purposes only.

Scroll to Top