Indeed I remember a demonstration in a gay disco (Khon Kaen around 2015, long closed) where condoms were inflated to the size of a head and then touched with a drop of lube (nothing happend) or a drop of oil (condom burst).
But recently in Cambodia, a friends asked to use baby oil as lube and we did, and the condom stayed intact. I didn't even think about oil damaging the condom.
From a chemical point of view, baby oil (be it based on mineral oil or vegetable oil) and silicon are similar. But I would never use WD-40, simply because it's not intended to be used on the human body and could contain harmful substances.