It probably is illegal if you share your garden because you can only be naked in private places. If you are naked and answer the door to someone that is not illegal as long as you do not step off of your property and sunbathing nude in your own private garden is also not illegal even if people can see you. If I were you I would sunbathe nude anymore as that man sounds like a bit of a pervert bringing you drinks and stuff. Probably just wants an eyeful.
You need to check your local laws in regard to your question as all states and city's vary on what is allowed in private areas.
If you live in England, Wales & Northern Ireland, in theory, it is not illegal to be nude anywhere, although you could be charged with a Public order offence e.g. Conduct likely to cause a Breach of the Peace. It is an offence for someone, male or female, to expose their genitals if they intended another person to see them and to be caused alarm or distress. There is also the common law offence of outraging public decency. If you deliberately annoy your neighbours you could be served with an Antisocial Behaviour Order. However, as your male neighbour appears to have no problem, I can see no reason why you should not continue sunbathing nude.
You can come and sunbath in my bed if youd like :)