Preheat your oven to 350°F (175°C).
In a large bowl, combine the mashed sweet potatoes, brown sugar, and melted butter.
Add the milk, eggs, vanilla extract, cinnamon, and nutmeg. Mix until smooth.
Transfer the sweet potato mixture to a greased baking dish.
In a separate bowl, combine chopped pecans and crushed graham crackers (if using).
Sprinkle the pecan mixture evenly over the sweet potato layer.
Bake in the preheated oven for 45 minutes or until the topping is golden brown.
Remove from oven and let it cool for a few minutes before serving.
Serve warm as a side dish.
Enjoy your delicious sweet potato casserole with pecans!