There is not enough official information to confirm this claim to be true. Despite being generally accepted by the public as fact, neither Ethan Slater nor Ariana Grande's team have made official statements confirming their relationship started as an affair. Other than a statement Slater's estranged wife Lily Jay made via Page Six, no official source indicates the relationship started from an affair.
There are timelines from Teen Vogue and Glamour that indicate the pair were very close prior to their respective separations from their partners, but nothing has been officially confirmed.