Hong Kong is not what I'd consider a casual city, but it is also a business-driven city. In better restaurants, smart casual (slacks and a nice shirt) are ok, although most of the patrons will be in business suits. In the best restaurants (Petrus, etc), jackets are required.

Having lived here for more than I decade, my view is that Hong Kong is kind of a funny place dress-wise. The concept of business casual has not been adopted here for offices, and it translates to restaurants in the business district (esp for lunch on weekdays). On the other hand, I have seen everything from T-shirts to dress suits on men at better restaurants. Petrus, mentioned above, is one of the few places where a dress code is actually enforced. There are a handful of other places where I think you would feel out of place if you did not have a suit or a least a sports jacket (The Mandarin Grill is one.) But I can think of dozens of restaurants in Hong Kong where the food is excellent and no one cares what you are wearing. A place like Nobu, which is wildly expensive, is not the kind of place where you would need to wear a jacket. And places like Tung Po Seafood are extremeley casual, plastic chairs and flurescent lighting (although not especially cheap.) So it is not always about price. And it appears you are not going to places like Petrus, IMO, jeans and a sweater will be fine where you would be going.

I think it is easy to pack a dress shirt, and you may prefer that to the sweater in some cases.

During the day for touring, Hong Kong is extremely casual. Wear whatever you are comfortable in.

i wore chinos and a decent shirt or polo and never felt out of place...

no shorts at dinner for sure or collarless shirts.

in better restaurants more might be required. casual is generally the accepted normal.