Why does the sun appear to rise and set on the north side of my house in the summer?

I have a house in Colorado, with the backyard side facing straight west (towards the Rockies). I noticed that in the summer the sun appears to set to the north of the western side of my house (and shadows are cast in a long diagonal from North to South). In the winter the sun appears to set to the south of that side (with shadows going the other direction). Since I am located in the Northern hemisphere, shouldn't the sun always set on the southern side of the western side of house? I am sorry but I have seen this enough times that I am a little confused.