Preheat the oven to 325 F. In a medium skillet, heat the sugar over medium heat. Do not stir! Let sugar melt and start to caramelize. You can shake or swirl the pan, but do not stir with a spoon. When sugar is a nice light brown color, carefully pour caramel into a glass loaf pan, tilting to make sure it coats the bottom and up the sides a little bit. Set aside.
In blender jar, combine condensed milk, milk, eggs, grated cheese, pecans and brandy. Process until very smooth. Pour over prepared caramel in pan.
Set the loaf pan inside a larger pan, such as a roasting pan. Place in the oven and then carefully pour enough boiling water into the roasted pan so that it comes two-thirds of the way up the sides of the loaf pan. Bake until a toothpick inserted in the center comes out clean, 1 hour to 1 hour, 15 minutes.
Cool and then chill at least 4 hours, but preferably overnight. Run a knife around the edges and then invert onto a serving platter, making sure you catch all the nice syrup. Serve chilled.