Preheat the oven to 350°F. Line a 9x13-inch cake pan with parchment paper and set aside.
With an electric mixer, beat the eggs and sugars until fluffy, about 3 minutes. Add the pumpkin puree, vanilla, milk, and oil. On low speed, beat until combined.
Sprinkle the flour, cinnamon, pumpkin spice, baking powder, baking soda, and salt over the egg mixture. Gently stir in the dry ingredients, then beat on low speed until combined.
Transfer to the pan. Bake for 30-35 minutes, or until a toothpick inserted into the center comes out clean.
Remove the cake from the oven once slightly cooled, and transfer to a wire rack to cool completely.
Meanwhile, beat the cream cheese, butter, sugar, vanilla, and pumpkin spice until creamy.
Pour in the heavy cream and beat on high speed until fluffy, about 3 minutes. Refrigerate until the cake cools.
Spread the frosting over the pumpkin cake. If desired, top with chopped nuts and drizzle with caramel.