...the dancing ants and stroke option is only available with live selections, like using the marquee tool or apple/control clicking a layer...
...when double clicking a layer you reveal the layer style dialog, at the bottom is the stroke option, ensure it is checked and also highlighted by selecting the name...
...controlling the color is done by clicking on the color fill type drop down menu, below that is a color square, click on it and you enter the color picker...
...to have the box transparent, turn the "fill opacity" down to zero, this is found top of the layers palette, underneath "opacity". It is also found in the layer style dialog (double click a layer), at the top of the list on the left hand side, select "blending options" at the very top...
...under "advanced blending" section you have the "fill opacity" and a number of other controls...
andrew