There's another way to approach this with a little less room for error...
Go to your channels list and create a new channel. Make a rectangular selection and fill it with white. Deselect. Now go up to the filters menu and select Blur->Gaussian Blur. Adjust the blur until you see faded round corners of the desired size. Now go up to your Image menu and select Adjust->Levels. Underneath the histogram, pull the arrows at the extreme right and left toward the center. You'll notice on the image that the rounded corners and edges get a brighter white tone. Adjust the arrows until the bright white reaches the edges but doesn't look aliased (jagged).
Once you've finished your new channel, Ctrl-Click the channel on the list to select it. Go back to your layers list and click on the layer you wish to modify. Underneath the Select menu, Transform the selection and resize to the desired dimensions and type Ctrl-Shift-I to invert the selection, then hit delete to remove the contents and you've got yourself a rounded image.
Hope this helps,
brendanc@icehouse.net