Several other people had some good suggestions, but I'm going to add a few more things that worked for us.
1 -- Make sure he's fed and rested and has gone to the bathroom before you go out in public. A hungry, thirsty, tired, or needing-to-pee kid gets restless and cranky quickly.
2 -- Talk about expectations before you go in. I used to do the following as I was parking the car outside a store or other public place. I'd ask, "Do we run in the store?" and they'd holler back, "NOOOOO!" "Do we take things off the shelves?" "NOOOOO!" "Do we scream and yell and throw fits?" "NOOOOO!" "Do we lick the freezer doors (or whatever ridiculous thing they'd done the last time we were in public)?" "NOOOO!" "Do we stay close to Mommy?" "YEEEESSSSS!" "Do we hold hands in the parking lot?" "YEESSSSS!" etc. They thought it was fun, and it reminded them of the rules.
3 -- Include them in the trip as much as possible. Yes, it may take you a bit longer, but it will be more enjoyable for you both. At the grocery store, I'd have my kids help me find certain items. Sometimes, they'd get their own list with pictures pasted on it (or simple words once they were reading) for items like apples, milk, etc. Sometimes, I'd let them help with the decisions -- red or green grapes, Rice Krispies or Cheerios, etc. If there were samples, we'd taste them. Your son can count items as you bag them -- 1, 2, 3 pears. You can point out the aisle numbers as you go down them -- "Here, aisle 4 -- this is where the peanut butter is." At the library, I'd let them pick a couple books first, and if I had to get anything for myself, I'd do it as quickly as possible. In restaurants, I'd bring a small book, toy, crayons, or something they could stay busy with at the table.
4 -- Keep the trip as short as possible.
5 -- Make sure he knows the plan for the day. I used to do this Dora-style (if he watches that show at all). "Today, we're going to have breakfast, then go to the grocery store, and then play at the park. Breakfast, grocery store, park. Breakfast, grocery store, park. Got it?" I know it sounds silly, but my kids always did much better when they knew exactly what to expect.
Most of all, hang in there, and stay patient. Age 3 is a tough year, and it does get better.