The exact growing conditions for olive production aren’t common in the US, so most of the production comes from California - west of Sacramento and south along the San Joaquin river. There are a lot of barriers in bringing specialty crops to market related to know-how and contracting sale of product, so even in other areas where growth may be possible it may be infeasible.
https://www.agmrc.org/commodities-products/fruits/olives
https://croplandcros.scinet.usda.gov/