How to Evaluate an AI Trading Tool Before You Trust It

The short answer

You cannot open a model and check whether it works. What you can do is interrogate the claims around it, and the questions below are ordered so the cheapest and most decisive come first.

Most tools fail at question two. That is not a scandal — it reflects how the industry reports results — but it does mean you can usually finish an evaluation quickly.

A note on what this page is not: we do not score tools, rank them or recommend any. This is a method you apply yourself, which is more durable than a verdict of ours that goes stale.

1. Which kind of tool is this?

An indicator computed from price, a model fitted to history, and a text generator are three different technologies, and the marketing word covers all of them. Settle this before anything else, because the right follow-up questions differ for each — which machine you are holding is the whole subject of our pillar.

A non-answer here is disqualifying. A vendor who will not say what their product computes is not going to be straight about anything harder.

2. Was it measured on data it had never seen?

This is the question. Everything else is secondary.

Performance on data a tool was developed against tells you about the fitting process, not about the tool. The clearest evidence we have read is a 2023 benchmark that evaluated fifteen state-of-the-art deep-learning models under one common framework:

"Our extensive experiments reveal that all models exhibit a significant performance drop when exposed to new data, thereby raising questions about their real-world market applicability."

All fifteen. That is the base rate you are arguing against when a vendor shows you a result, and it is why "how did it do on unseen data?" is worth more than any feature comparison.

Note the scope honestly: that benchmark covers short-horizon prediction from limit-order-book data, not every model and market. It is not a proof that all tools fail. It is a strong reason to treat in-sample results as uninformative.

3. How many versions were tried before this one?

A good-looking result is easy to produce by trying many configurations and keeping the best. The number tried is what determines how much the survivor means, and it is almost never disclosed.

Ask. A vendor who can answer has thought about the problem; why good backtests fail live explains why the number matters more than the curve.

4. Who operates it, and are they authorised?

A matter of public record, and quicker to check than reading a feature list. ESMA's own investor factsheet makes the point that in the EU, providing investment advice is a regulated service that requires authorisation, and points investors at the register of authorised firms.

This tells you what recourse exists if something goes wrong — which is a different question from whether the tool is any good, and people routinely conflate them. What regulators say covers the detail.

5. How would I know if it were wrong?

A tool whose output cannot be falsified cannot be evaluated at any price.

For a signal product this is concrete: is there a stated invalidation? For an analysis product: does it commit to anything specific enough to be checked later? Vague directional commentary is unfalsifiable by construction, which is why it is so popular.

How signals are generated sets out the parts a signal needs to be checkable at all.

6. What does it do when it does not know?

Good systems have a way of declining. The failure ESMA warns about is precisely that these tools do not naturally express uncertainty:

"AI tools can generate advice that could be inaccurate or misleading and that may result in poor investment decisions and significant financial losses."

A tool that produces an answer of equal confidence to every question, including ones outside its competence, has no uncertainty handling at all. Ask it something unanswerable and watch.

7. What is it allowed to do on its own?

There is a large difference between a tool that suggests and a tool that executes. The second is a delegation of discretion, and it deserves scrutiny of a different order — including how it is stopped. For automated execution specifically, what expert advisors do covers the documented mechanics.

8. What happens to my data and my money?

Where does your trade history go, who else sees it, and does the tool ever touch funds? A tool that only reads public prices carries a different risk profile from one connected to your account.

Reading the answers

Three patterns are worth more than any individual response.

Specificity. Good answers are boring and concrete. Evasive answers get more impressive as they get vaguer.

Who the result belongs to. "Our users have seen" is not a measurement. Be aware that a published figure usually reflects both the tool and the selection of which results to publish.

What happens when you ask about failure. A vendor who can describe when their tool does badly has tested it. One who cannot imagine the question has not.

We do not repeat vendors' performance figures anywhere on this site — the reasons are in our editorial policy, and the limits of all of this are in the risk disclosure.