LMS Vendor Evaluation: The RFP Questions That Reveal What Demos Don't Show

Quick answer: An LMS demo shows capability under ideal conditions. A request for proposal forces disclosure about conditions, volume, cost and failure behaviour, which is where platforms actually fail. The questions that separate vendors cover migration effort, administrative workload, integration depth, renewal pricing, support reality and product direction. Written answers should end up in the contract rather than in a folder nobody opens again.

An LMS demo is a controlled performance. The tenant holds forty users and clean data. There is no fifteen-year completion history, no duplicated employee records from an acquisition, no location hierarchy that three different teams defined differently. The solutions engineer has run that click path several hundred times.

None of that is dishonest. It is what a demo is.

The problem arrives eighteen months later, when someone on your team is exporting a CSV every Friday, editing it by hand, and re-importing it because bulk enrollment cannot handle your site structure. That capability was demonstrated. It works. It does not work at the shape and scale of your organization, and a demo was never going to reveal that.

Why do LMS demos look so good?

Three things are true of every demo environment, and all three favour the vendor without anyone bending the truth.

The data is clean. Demo tenants contain synthetic records built to behave. Your environment contains fifteen years of accumulated decisions and several hundred people who exist twice.

The volume is small. Enrolling eight learners is a different operation from enrolling four thousand across sixty sites with location-specific due dates.

Nothing after the sale is visible. Implementation slippage, support responsiveness in month fourteen, renewal pricing, whether the roadmap ships. A sixty-minute call cannot surface any of it, and all of it is what you are actually buying.

Keep the demo. Stop treating it as the evidence.

What should you decide before writing the RFP?

Write requirements as workflows, not features

A feature can exist and still fail the workflow. Two vendors may both answer yes to custom reporting, but one offers an administrator report builder and the other quotes professional services. Both may support external learners, but only one matches the identity and delegated-administration model you need.

The practical difference: "supports blended learning" cannot be failed. "Enrolls 400 people across three sites with different due dates, then changes one site's date post-launch" can. Write the second kind.

Weight the scoring rubric before you read a single response

Teams that assign weights after reading responses reliably discover that the criteria favouring their preferred vendor were the important ones all along. Nobody does this cynically. It is ordinary anchoring, and the only defence is committing the weights to writing before the anchor exists.

Agree them with HR, IT, L&D and procurement together, before the first vendor call.

Reserve mandatory status for things that are actually mandatory

Too many must-have items make comparison meaningless. Mandatory should mean a legal obligation, a data protection requirement, an essential workflow, or a technical dependency that cannot be worked around. Everything else is a preference with a weight.

The 41 LMS RFP questions that reveal the most

Send these in writing and require written answers. A vendor who answers verbally but declines to put it in the response document has told you something useful.

Data, migration and exit (6 questions)

Migration is where implementations quietly fail, and it is the area vendors most often answer in generalities.

1. Which record types transfer natively from our current system, and which do not? Provide a list, not a reassurance.
2. After migration, is historical completion a native record inside the platform, or an imported flat file?
3. Does migrated data support recertification logic and expiry calculations, or only display?
4. Who performs the migration, and how many hours does it require from our team? Split the estimate between IT and L&D.
5. If we terminate, what is the export format, what does it cost, how long does it take, and does it include historical completion records?
6. How long do you retain our data after termination, and what is the deletion process?

Question five is the one almost nobody asks, and it is the most revealing question in the document because it is the only one with no upside for the vendor. If you operate in the EU, the GDPR right to data portability gives you a floor, but a contractual export commitment is faster to enforce than a regulatory one.

Administration and the Tuesday afternoon test (5 questions)

Somebody will operate this system every week for four years. Build the evaluation around that person's day rather than around executive dashboards.

1. Walk through enrolling 400 employees across three locations with location-specific due dates, live in the platform.
2. Change the due date for one location after launch. How many steps?
3. Which of these actions require a support ticket rather than self-service?
4. Which require an administrator with technical skill versus a coordinator?
5. If a bulk action is applied in error, how is it reversed, and at what granularity?

The reversal question is consistently underrated. A system that makes bulk actions easy to apply and hard to undo generates a large volume of quiet manual work that never appears in any business case.

Integrations, measured by depth rather than logos (5 questions)

An integrations page is a wall of logos. A logo confirms a connection exists in some form. It says nothing about what flows, in which direction, how often, or what happens when it breaks.

1. For each integration we need: which fields sync, in which direction, at what frequency?
2. Is the sync real-time, scheduled, or manually triggered?
3. When a record fails validation, does it queue, drop silently, or halt the batch? Who is notified?
4. Is the integration built and maintained by you, by a partner, or does it require middleware we license separately? Is it in the quoted price?
5. Can you name a customer of similar size running this specific integration, and may we speak with them?

A vendor who produces that named reference has a working integration. A vendor who substitutes a general reference has answered a different question.

Reporting and whether it answers your real questions (5 questions)

1. Demonstrate live: which employee groups have the lowest participation, segmented by shift and location?
2. Demonstrate live: what did we spend per completion by business unit last quarter?
3. Can we build custom reports ourselves, or is that a professional services engagement?
4. Is there raw data export or API access to underlying records, and are there call limits?
5. How long is historical data retained at full granularity before aggregation or archiving?

Add one more if executives are in your reporting audience: can reports be scheduled and delivered to people who will never log in? That capability often decides whether your reporting is read at all.

Pricing and renewal behaviour (6 questions)

1. How is a user defined and counted: registered, active, or licensed? What is the measurement window?
2. What happens in a month where seasonal headcount spikes above contracted volume?
3. What is included in the base price, and what is a separately priced module? Provide the full list.
4. Is there a cap on annual renewal increase written into the agreement?
5. What has your average renewal increase been across the last three years?
6. Provide a three-year total cost of ownership including implementation, migration, integration, training, and our internal administrative time.

Three definitions cause most of the surprises: how a user is counted, what happens when headcount spikes, and what a renewal is permitted to increase by. Get those three in writing and the rest of the pricing conversation becomes manageable.

On question five, a vendor may decline to answer. The decline is informative and should be recorded in your scoring notes.

Implementation and support reality (5 questions)

1. Who specifically runs our implementation? Name the role, ideally the person, and disclose their concurrent projects.
2. Provide a timeline with our dependencies marked, so we can see how much of the schedule depends on our own team finding time.
3. What are the support tiers, the guaranteed response times by severity, and the difference between a response and a resolution?
4. What are support hours across our operating time zones, including sites outside headquarters?
5. What happens when our customer success manager changes, and how is handover documented?

That last question predicts year two more accurately than anything in the demo. Ask a reference customer how many customer success managers they have had.

Product direction (4 questions)

Roadmaps are marketing. Shipping history is evidence.

1. What shipped in the last twelve months, with release dates?
2. What was deprecated or sunset in the same period, and how much notice did customers receive?
3. How do customers influence priorities, and what was the most recent customer-requested feature to ship?
4. Is the product we are buying on the same platform you are investing in, or is a successor product in development?

Security, accessibility and standards (5 questions)

1. Provide the current SOC 2 Type II report itself, not a summary page. What is the report date and the observation period?
2. Where is data hosted, and can residency be guaranteed for each jurisdiction we operate in?
3. Provide a current VPAT. Which WCAG version and conformance level do you meet, and where are the known gaps?
4. Which content standards are supported and at what conformance level: SCORM 1.2, SCORM 2004, xAPI, cmi5? Is a learning record store included or expected separately?
5. Will you test a sample of our actual content packages, not a reference package?

Every vendor has accessibility gaps. The ones who name them are easier to work with than the ones who claim none. If you have public sector obligations, confirm Section 508 alignment explicitly rather than assuming WCAG conformance covers it.

How do you translate a demo claim into an RFP question?

A demo statement asserts a capability. The useful version specifies conditions, volume and failure behaviour.

What should a scored demo script contain?

Give shortlisted vendors the same realistic script and enough background to prepare. Keep it structured enough to compare and loose enough to allow discovery.

The exception that reveals most: if required training changes when an employee moves departments, ask the vendor to process that move live. Watch whether the old assignment remains, disappears, or needs manual review. A perfect new-hire workflow tells you far less.

How should you score vendor responses?

Score against the rubric you weighted before the process began. A weighted score supports comparison but should not make a close decision look mathematically certain.

These weights are an illustration. An organization replacing a compliance LMS should weight governance and records more heavily. A customer academy should prioritise external identity, branding and audience administration.

Three practices make scoring more honest. Score each section independently rather than forming an overall impression and distributing numbers to match. Have each function score the sections it owns, then reconcile, treating large divergences as the most interesting data in the process. And require one sentence of written justification for any score at the extremes.

Keep mandatory failures visible instead of averaging them away. Watch also for the vendor who scores well everywhere and is exceptional nowhere: sometimes a balanced platform, more often a proposal team that knows how RFPs are scored.

What are the red flags during an LMS evaluation?

• Every requirement answered yes, with no delivery method identified.
• Roadmap functions presented as current capability.
• Configuration described as custom development only after pricing.
• A report that cannot be built during the evaluation.
• Security questions redirected to marketing material.
• An implementation plan with the client responsibilities left blank.

A vendor does not need to meet every preference. A clear limitation is more useful than an ambiguous promise, because it lets your team decide whether the tradeoff is acceptable.

What should you ask reference customers?

Every vendor supplies references and every supplied reference is happy. Ask questions a happy customer will still answer truthfully.

Insist on shape-matched references first: similar headcount, similar industry, similar ratio of desk-based to deskless staff, and live for more than eighteen months. A customer three months post-launch is still inside the honeymoon and still has the implementation team's full attention.

1. What took longer than you expected?
2. What do you do manually that you assumed the platform would handle?
3. How many customer success managers have you had?
4. What did your renewal increase look like?
5. Which configuration became difficult to maintain?
6. If you were starting again, what would you ask that you did not ask?

The last question routinely produces better information than the rest of the call combined. Where you can, find a customer who is not on the vendor's reference list; industry peer groups usually surface one, and the conversation is materially different.

How do you turn RFP answers into contract terms?

This is where most evaluations lose their value. A team runs a rigorous process, collects detailed written answers, selects a vendor, and then signs the vendor's standard agreement, which references none of it.

Attach the RFP response as an appendix and pull specific commitments into the body of the agreement:

• Implementation timeline and named resources belong in the statement of work.
• Support response times belong in an SLA with a remedy attached, not a marketing page.
• Renewal increase caps belong in the commercial terms.
• Integration scope belongs in writing with the synced fields listed.
• Data export commitments belong in the termination clause with format, cost and timeline specified.

The working test is one sentence: if a vendor will not put an answer in the contract, it was never an answer.

Keep the demo script afterwards. Your project team should be able to repeat the priority learner, manager, administrator, data, exception and reporting workflows in the configured environment before launch.

How long should an LMS evaluation take?

Roughly three to four months for a decision your organization will operate inside for the next four years.

When the timeline compresses, the stages that get cut are almost always references and contract negotiation. Those are precisely the two that protect you.

The question underneath all of this

An RFP is not really an interrogation of a feature set. It is a test of how a vendor behaves when asked something specific and slightly inconvenient.

Some vendors answer the hard questions plainly, including the parts where the answer is no. Others restate the question as a benefit. The first group is not automatically selling the better platform. They are showing you how they will communicate during a difficult implementation, a missed deadline, or a renewal conversation, and you will have far more of those than you will have demos.

Buy the platform that fits the work. Weigh heavily how each vendor behaved while you were finding out.

Frequently asked questions

How many vendors should be in an LMS RFP?

Three to five for the written RFP, narrowed to two or three for scored scenario demos. Fewer than three removes comparison leverage. More than five produces a volume of response documentation that no evaluation team reads carefully, which defeats the purpose.

What is the difference between an RFI and an RFP for LMS selection?

An RFI is exploratory, used to build a shortlist from a wide market, asking broad questions about capability and fit. An RFP is comparative, used to select from a shortlist, asking specific questions whose answers can be scored and written into a contract. Run an RFI first when the market is unfamiliar to you, and skip it when it is not.

What is the single most revealing LMS RFP question?

How would we get our data out if we left, in what format, at what cost, and does it include historical completion records? It is revealing because it is the only question with no upside for the vendor, so the manner of the answer carries as much information as the content.

Should we ask for a sandbox before deciding?

Yes, and load it with a sample of your own data rather than the vendor's. A sandbox populated with clean demo records reproduces the original problem. Even a few hundred of your real employee records, with the messy hierarchy intact, will surface issues no scripted walkthrough will.

How do we evaluate an LMS for deskless or frontline workers?

Test on the device and connection those employees actually have, not on a laptop. Ask about offline behaviour, shared-device login and logout, session timeout, mobile completion tracking and language support. Then ask each vendor to name a reference customer with a comparable frontline population, because design assumptions built for desk-based workforces do not transfer.

Who should be on the LMS evaluation team?

At minimum: L&D as process owner, the administrator who will operate the system daily, IT for security and integration, procurement for commercial terms, and one business stakeholder from a function that will use it. The administrator is the member most often omitted and the one whose experience most reliably predicts whether the platform succeeds.

What if a requirement depends on a roadmap item?

Decide whether the current product is acceptable without it. If the answer is no, the buying decision should not assume an unconfirmed future date. If the answer is yes, score the product as it exists today and treat the roadmap item as upside rather than as a criterion.

For the broader capability areas to cover before writing requirements, see the LMS features evaluation guide. To convert vendor answers into named implementation responsibilities, the LMS implementation checklist is a practical next step, and the LMS security checklist should be reviewed by your technical and security teams rather than treated as something a product demo can validate. If you would like Trainery to answer this question set directly, start with the corporate learning management system overview and bring your own workflows to the session.