Preheat the oven to 350 F. Grease a 9-inch loaf pan (or a 9-inch round cake pan) and set aside. Whisk together the flour, baking powder, salt and baking soda in a medium bowl and set aside.
Combine sugar, lemon, and orange zest in a large bowl or the bowl of an electric mixer, and rub together with your fingers to release the citrus oils into the sugar. Once the sugar is fragrant, add the eggs and beat with an electric mixer for 3 minutes. Add the olive oil and continue to beat until pale yellow in color. Add the orange juice and extract, and beat 1 minute more to incorporate.
Add the dry ingredients into the wet, and mix by hand just until completely incorporated. Pour into the prepared loaf pan, and bake 40 minutes, or until a tester inserted in center comes out clean.
Turn out from the pan onto a wire rack and cool completely before serving.