If it's a permanent or semi permanent color it's OK to go swim in a water or pool that has chlorine for the disinfecting purposes. If you used a temporary hair die you should be aware that every time you wash your hair it will fade the color. See hair color direction on the box.
You need to fully wet your hair before getting in the pool. If you dyed it 2 days ago, it should be okay time span so you won't have any real color damage. If you completely wet your hair with tap water that will lessen the amount of chlorine it absorbs and lessen the chances of your color fading. You need to rinse your hair out with tap water when your done swimming too.