Strategists at Barclays said that a rally at the end of the year would likely need oil prices to lower first.

Fading investor momentum has left quant funds as the stock market's main support, and that leaves a year-end rally highly dependent on oil prices, according to Barclays.

While appetite in Wall Street is deteriorating and retail sentiment turned its most bearish so far this year, systematic funds, which are run by computers using a predefined algorithm, saw their exposure to stocks rise to close to peak levels, a team at the London-headquartered bank, led by Emmanuel Cau, head of European equity strategy, wrote in a Wednesday note.

Stock prices generally decline in September, but the historically best-performing final quarter of the year and the upcoming midterm elections should help to give them a boost, they said.

"Although FOMO remains palpable, erratic price action suggests a lack of conviction," the strategists wrote. "This time indeed, a year-end rally and renewed rotation into Europe likely require lower/stable oil prices, which is anyone's guess."

If the U.S. and Iran were to agree on a peace deal, it would help to raise both stocks and bonds, they said, but the bond market in particular appears to be pricing in oil staying heightened for longer. It comes after the yield on the 30-year U.S. Treasury BX:TMUBMUSD30Y hit a 24-year high at 5.617% on Tuesday and the 10-year's BX:TMUBMUSD10Y yield reached close to its 19-year peak.

U.S. equity inflows have slowed, with outflows seen in four of the previous five weeks, and at the same time, European inflows have also eased as hedge funds exited their long positions when oil prices spiked, according to Barclays.

September also saw narrowing market breadth, with funds heavily exposed to the semiconductor, capital goods, utility and materials sectors, while software, healthcare, luxury and other defensive stocks were under-owned, the strategists said.

"Energy flows have picked up only marginally despite higher oil prices," they wrote. "But if oil and rates ease, capex leadership could re-emerge and consumer stocks may be due for a tactical catch-up."

-Nora Redmond