In its statement of objections, the OFT has deemed the practice harmful to consumers because it restricted the competitive process, leading to higher prices.
The OFT said it had warned the supermarkets about the practice previously and that they understood their actions might be anti-competitive.
Sean Williams, OFT executive director, said: "This is a very serious case. We believe supermarkets have been colluding to put up the price of dairy products. Consumers have lost out to the tune of hundreds of millions of pounds.
"This kind of collusion on price is a very serious breach of the law. Businesses should understand that where we find evidence of this kind of anti-competitive activity we will use the powers at our disposal to punish the companies involved and to deter other businesses from taking such actions."
However, the OFT cannot decide if the Competition Act has been breached until it reviews the supermarkets' responses to the statement of objections.
The Competition Act 1998 prohibits agreements, practices and conduct that may have a damaging effect on competition in the UK.
Last year, the OFT decided to refer the grocery sector to the Competition Commission. It began the investigation last May and has until next year to come up with a decision, although it hopes to complete its review well before then.
The watchdog said at the time that there was evidence that the buying power of big supermarkets could distort competition and that it was also concerned about planning rules and supermarket land holdings.