Prepare a 9-inch square pan by lining with wax paper.
Clip candy thermometer to 3-quart saucepan. Combine condensed milk, whole milk, sugar and butter over a low heat, stirring until the sugar dissolves.
Bring mixture to a steady boil for about 20 minutes, stirring frequently and scraping the sides and base of the pan to prevent burning. The mixture should reach 235-245°F (soft ball stage) on candy thermometer.
Remove from the heat and let it cool for a few minutes.
Add remaining ingredients and beat the mixture at high speed until it thickens and pulls away from the sides of the pan, about 8-10 minutes.
Pour fudge into the prepared baking dish and spread evenly. Let cool completely, about 2 hours, before cutting into 1-inch squares.
Store in an airtight container.