Place popcorn into a large bowl. Shake the bowl and toss gently until unpopped kernels sink to the bottom. Discard unpopped kernels.
Combine brown sugar, corn syrup, and butter in a saucepan over medium-high heat. Cook, stirring constantly, until mixture reaches the soft-crack stage. To test, the temperature should be between 270 to 290 degrees F (132 to 143 degrees C) and a small amount of syrup dropped into cold water should form hard but pliable threads.
Remove from the heat. Carefully pour condensed milk into the saucepan and stir until smooth.
Pour 1/4 of the caramel at a time over popcorn, stirring well after each addition.
Let cool for at least 10 minutes before serving.