Project Canvas has estimated its total costs over a four-year period at £115.6m in a submission to the BBC Trust.
Five claims its final decision depends on the outcome of a review started by director of strategy Charles Constable. It is still likely to provide content to the platform.
Constable said: "We continue to support the objectives of Project Canvas and despite withdrawing our interest in the venture we believe it will be a critical part of our strategy for reaching consumers in the future."
Project Canvas, which is not yet formally incorporated, has been seeking equal investment from all seven partners, which include the BBC, ITV, BT, Channel 4, TalkTalk and Arqiva.
It is not yet clear how Five's costs would be shared between the remaining six partners.
The BBC has been required by the BBC Trust to ensure its involvement does not exceed its estimated costs by 20% over a five-year period, as a condition of the Trust approving the project in June.
A spokeswoman for Project Canvas said it was now looking at the implications of Five's decision.
The platform was expected to launch before Christmas.