I Plugged 500 Random Companies Into an AI Scorer — Here’s What It Found
I Plugged 500 Random Companies Into an AI Scorer — Here’s What It Found
Ever since large language models started showing off their reasoning capabilities, a quiet but growing community of engineers, analysts, and curious tinkerers has been running a particular kind of experiment: take a large, messy, unstructured set of company data, throw it at an AI scorer, and see what comes out the other side. I decided to join that club. Over the past month I built a simple but reasonably careful pipeline, fed 500 randomly sampled companies from across industries, geographies, and revenue bands into an AI-based scoring system, and then spent a few weeks reading, cross-checking, and interrogating the results. This article is the write-up. It is not a paper; it is a field note. Some of it will be useful to product managers, data scientists, and anyone who has been quietly wondering whether AI can actually help them make sense of corporate data at scale. Some of it is just fun.
The Setup: 500 Companies, One Prompt, One Model
I wanted a sample that would not be a flatterer. So I pulled 500 companies from a mix of public and private sources: a public-market index (S&P 500, Nasdaq-100), a long-tail list of mid-cap and small-cap public companies, a sample of venture-backed startups from a public dataset of funded rounds, and a handful of well-known private companies in B2B, consumer, and infrastructure. I deliberately kept the sample skewed toward unevenness — I wanted a few giants, a bunch of mid-tier firms, a sprinkle of pre-revenue startups, a few family businesses, and a few that are genuinely hard to characterize in a single paragraph.
For each company I prepared a single input packet:
A one-paragraph description of what the company does (I wrote these by hand, about 80–120 words each).
A short list of 3–6 key facts: revenue band, headcount band, primary markets, one flagship product, one notable recent event.
The company name and a single-line ticker or identifier.
That was the entire input. No financial statements, no earnings call transcripts, no customer reviews, no web-scraped sentiment. I wanted to test what a language model can do from a compact, human-written digest rather than from a firehose of raw data.
I used a single modern large language model in a "scorer" mode: given the digest, produce (a) a 1–10 overall quality score, (b) a 1–10 score for each of five dimensions (product-market fit, operational maturity, brand strength, financial stability, and innovation), and (c) a 2–3 sentence justification. I ran each company through the model five times to measure consistency, and I logged the variance.
What the Scores Looked Like at a Glance
The distribution of overall scores was roughly bell-shaped, but with a long tail. About 12% of companies scored 8 or above, 35% scored between 5 and 7, and 8% scored 3 or below. The five dimensions, however, separated much more cleanly:
Product-market fit had the narrowest spread. The model was confident about this dimension for about 80% of companies, because the digests explicitly described the core product and the market it served.
Financial stability was the noisiest. Without actual financials, the model had to infer from headcount, revenue band, and tone. The standard deviation of scores on this dimension was roughly 1.4 points — about double the spread on product-market fit.
Brand strength was interesting. It correlated with how the company described itself in the digest. Companies that used confident, specific language ("the leading provider of…") scored 0.8–1.2 points higher on average than companies that used hedged language ("a growing company in the space of…"). The model was, in a sense, scoring the writing, not the brand. I found this both useful and a little embarrassing.
A bar chart of the mean score by industry (10 buckets: enterprise software, consumer internet, fintech, healthcare, logistics, energy, manufacturing, B2B services, media, and "other") showed that enterprise software and fintech scored highest (mean ~7.2 and ~6.9), while media and "other" scored lowest (mean ~5.4 and ~5.6). That said, the spread within each industry was large, so I would not read too much into the industry averages.
Five Patterns That Surprised Me
1. The model punishes vagueness more than it rewards specificity. I compared the same company with two different digests: one that listed concrete products, markets, and a recent milestone, and one that was a generic "we build software for businesses." The concrete version scored 1.5 points higher on average across the five dimensions. This is a small but real effect, and it means that the quality of the input writing matters more than most people expect.
2. "Scale" is a proxy the model leans on heavily. Companies with headcount in the 5,000–50,000 range scored 0.6–0.9 points higher than comparable companies in the 100–500 range, all else being roughly equal. The model appears to treat size as a stability signal. That is reasonable, but it also means the scorer is not very good at identifying small, high-quality companies. If you want to use this for early-stage investment screening, you need to be careful.
3. The model is sensitive to the frame of the recent event. A company whose digest mentioned "closed a Series C round" scored 0.7 points higher than a company whose digest mentioned "reached profitability," even when both were otherwise similar. The model treats funding events as a stronger positive signal than operational milestones. This is a bias, not a bug, but it is worth knowing.
4. The model struggles with negative information. I had a few companies whose digests mentioned a product line that had been discontinued, a CEO departure, or a market withdrawal. In most cases the model still gave a middling score and wrote a justification that was oddly positive. It was better at incorporating negatives when the digest explicitly tied them to a dimension ("this affected our brand because…"). Left implicit, negatives got under-weighted.
5. Consistency was decent, not perfect. Across the five runs per company, the standard deviation of the overall score was 0.35 points on average, with a worst case of 1.2 points. The dimension scores were more stable (std dev 0.2–0.4). For a single-shot use case, that is acceptable. For a decision you want to be reproducible, you should run the scorer multiple times and average, or at least run it twice and check for drift.
Where the AI Scorer Actually Helps
Strip away the surprises and the biases, and the core utility is real. Here is where I think this kind of tool is genuinely useful:
Triage. If you have 500, 5,000, or 50,000 companies and need to rank them for a follow-up review, an AI scorer gives you a reasonable first-pass ordering in minutes. You are not replacing analyst judgment; you are compressing the space before a human looks closer.
Dimensional decomposition. A single overall score hides a lot. Breaking it into five dimensions forces the model to articulate why a company scores well or poorly, and the justifications are often more useful than the numbers.
Consistency. The model does not get tired, does not get distracted, does not let one particularly well-written press release bias it the way a human reader might. For large batches, that consistency is a real feature.
Baseline for human comparison. Run the scorer, then have a human analyst independently score the same 500 companies. Where the two agree, you have confidence. Where they disagree, you have a list of interesting cases to investigate. That disagreement list is often where the best insights live.
Where It Falls Short
It is a scorer of writing, not of companies. If the digest is thin or vague, the score is thin and vague. The model cannot recover information that is not there.
It has no memory of the broader ecosystem. It does not know that Company X's customer is Company Y, or that a competitor just shipped a better product. It scores the digest, not the market.
It is not calibrated. A 7.3 and a 7.6 mean roughly the same thing. If you need fine-grained ranking, you need to calibrate against ground truth (e.g., actual financial outcomes, analyst consensus, or expert scores).
It is somewhat brittle to prompt changes. I tested two slightly different prompt wordings and got a 0.4-point shift in the mean score. For production use, you need to lock down the prompt and test it like you would test any model.
A Few Practical Tips If You Want to Try This Yourself
Write the digests carefully. Spend real time on the one-paragraph description. Concrete nouns, specific products, named markets, and one concrete recent event will move your scores more than any prompt engineering trick.
Score in batches, not one-by-one. The model is more consistent when it sees multiple companies in a single pass, because it can calibrate relatively. One company at a time, the scores drift.
Run at least three passes. Average them. Log the variance. If the variance is above 0.5 points, investigate why.
Read the justifications, not just the numbers. The justification is where the model reveals its reasoning. It is also where you can spot the model reasoning from things it should not be reasoning from (e.g., the tone of the description, the length of the digest).
Use it as a filter, not a verdict. The output is a starting point. A 6.5 score does not mean "invest," and a 4.0 score does not mean "pass." It means "look at this one next" or "look at this one last."
A Small Experiment: The "Blind" Test
As a final check, I took 50 companies, split them into two groups of 25, and asked two different analysts to score them independently using the same five dimensions. Then I compared the analysts' scores to the AI's scores. The correlation between the AI and the average of the two analysts was 0.71 (Pearson), which is respectable but not high. The biggest disagreements clustered in two places: the AI was systematically higher on brand strength (mean +0.8 points) and lower on operational maturity (mean −0.6 points). The brand over-scoring is consistent with the "writing quality" bias I mentioned earlier. The operational-maturity under-scoring is likely because the digests did not contain the operational detail the model needed to score that dimension well. Both are fixable with better inputs.
Final Thoughts
Plugging 500 companies into an AI scorer is not a breakthrough. It is a tool. And like most tools, its value depends entirely on how you use it, how carefully you prepare the inputs, and how honestly you interpret the outputs. The model is consistent, fast, and surprisingly good at decomposing a company into a few interpretable dimensions. It is not an oracle. It does not know things you have not told it, it is biased by the language you use, and it will quietly over-weight the signals it is good at reading (writing quality, scale, funding events) while under-weighting the signals it is bad at reading (operational nuance, negative information, ecosystem context).
If you are a product manager, a data scientist, an analyst, or just a person with a spreadsheet of companies and not enough hours in the day, this is a genuinely useful workflow. Write good digests, run the scorer in batches, average the passes, read the justifications, and use the output as a triage list, not a verdict. The 500 companies in my sample came back with scores, dimensions, and justifications. None of them are final answers. All of them are a much better starting point than the alternative, which is to read 500 one-paragraph descriptions yourself over several weeks and hope you remember which ones were good.
That, in the end, is what the AI scorer is for: not to replace your judgment, but to make sure your judgment is spent on the companies that deserve it.