Position a rack in the center of the oven and preheat the oven to 325°F. Lightly grease a 9" x 13" pan with non-stick cooking spray.
Mix ingredients together. In a large mixing bowl, stir together flour, brown sugar, sugar, baking soda, salt, cinnamon, ginger, and nutmeg until a crumbly mixture is formed.
Add the apples and nuts. Stir in with wooden spoon or spatula, the apples and pecans until a thick, smooth batter is formed, about 3 minutes.
Bake the cake. Spread the batter in the prepared pan smoothing the top with a spatula. Bake the cake for 45 minutes until a toothpick inserted into the center comes out clean, or with just a few moist crumbs. Remove the cake from the oven and place it on a wire rack to cool while you prepare the frosting.
Make the frosting. In a large saucepan, melt the butter over medium heat. Stir in the brown sugar and salt and cook, stirring, until the sugar begins to melt and the mixture becomes fairly smooth. Stir in the milk and bring to a boil. Remove from the heat and allow to cool for 10 minutes. If your saucepan isn't big enough, you may need to transfer to a large mixing bowl before adding the sugar in the next step.
Stir in the confectioners' sugar into the saucepan, then add the vanilla extract. Whisk until everything is thoroughly combined. The frosting can set quickly, so be sure to stir quickly!
Pour the warm frosting onto the cake(s), spreading it over the entire surface.