The rocks get warm in the sun no matter where the wind is coming from. But many fewer fish are caught when it's not the RIGHT wind. Therefore I think the reason it's a hot spot when the right wind is blowing, is because the right wind is blowing...and trapping schools of bait in front of my favorite rock.
