Preheat the oven to 350°F (175°C).
Line the bottom of a 9-inch round cake pan with parchment paper and thoroughly grease the sides and parchment paper. A springform pan works especially well.
Pit the peaches and slice them thinly into half-moon shapes.
Arrange the peach slices in a spiral pattern across the bottom of the prepared pan, starting from the center and filling any gaps. Set aside.
In the bowl of a stand mixer fitted with the whisk attachment, beat the butter, granulated sugar, and brown sugar on medium-high speed until light, fluffy, and pale in color.
Scrape down the sides of the bowl and add the vanilla extract.
Add the eggs one at a time, beating well after each addition until the mixture is smooth and fluffy.
In a separate bowl, whisk together the flour, baking powder, and salt.
Add half of the dry ingredients to the butter mixture and mix just until combined.
Add the milk and yogurt, mixing until incorporated.
Add the remaining dry ingredients and mix until no streaks of flour remain. Do not overmix.
Carefully spread the batter over the peach layer, smoothing it evenly to the edges without disturbing the fruit arrangement.
Bake for 30–40 minutes, or until a knife or toothpick inserted into the center comes out clean.
Allow the cake to cool completely in the pan.
Run a knife around the edge of the pan and carefully invert the cake onto a serving platter.
Remove the parchment paper and serve with the peach side facing up.