Finding the shortest path between two points within a (simple) polygon can be efficiently solved by using the funnel algorithm. For more details, see blog 1, blog 2, Paper 1 and Paper 2.
A good implementation in Python is available at GitHub margaeor