Chinatown for dim sum, and just to see the sites.
Japan town for food.
Little Italy for food, the Stinking Rose and Steps of Rome are good
Nothing much in Castro..
Metrion is good as well as Market Street Plaza.
If you have kids, the Exploritorium is a good location to have a picnic, and have them play with scientific gadgets.
Fishermans Wharf, hike around Angel Island, picnic at the top of the island ($12), or schedule a tour of the Prison.
All locations are located near public transit and the bart system. You can even go over to the East Bay and visit Albany Aquarium, and Berkeley..Zachary's Pizza on College Ave is a great place for Chicago, deep dish pizza. You can also visit the Berkeley Marina where there is one of the longest fishing piers on the West Coast (used to be 1 mile long), and there's always people out there flying stunt kites, giganitic kites, etc.
Enjoy your trip.