We consider a smooth compact manifold with boundary, $M$, embedded in a smooth manifold of the same dimension on which an amenable group $\Gamma$ acts by isometries. We do not assume $M$ to be invariant under $\Gamma$. This results in a {\em partial action} of $\Gamma$ on $M^\circ$: For $g\in \Gamma$ we let $M^\circ_g = g(M^\circ)\cap M^\circ$ and obtain diffeomorphisms $g:M^\circ_{g^{-1}} \to M^\circ_g$. We assume that any two images of $\partial M$ under $\Gamma$ either coincide or are disjoint and that only finitely many lie in $M$. The spherical blow-up of these images of $\partial M$ in $M$ yields a manifold $Y$ with boundary consisting of finitely many components. Moreover, $Y$ inherits a partial action of~$\Gamma$. We can then define the $C^*$-algebra $\mathcal A=\overline{\Psi_\Gamma(Y,\partial Y)}$ of operators on $L^2(Y)\oplus L^2(\partial Y)$, generated by the algebra $\Psi(Y,\partial Y)$ of operators of order and type zero in Boutet de Monvel's calculus on $Y$ and partial isometries associated with the partial action. Denote by $\Sigma=\overline{\Psi(Y,\partial Y)}/\mathbb K$ its symbol space. If the partial action of $\Gamma$ on Prim$(\Sigma)$ is topologically free, we find a criterion for the Fredholm property of the operators in $\overline{\Psi_\Gamma(Y,\partial Y)}$. Moreover, we obtain the classification of the elliptic elements in $\overline{\Psi_\Gamma(Y,\partial Y)}$ modulo stable homotopies: For $\mathcal A_0= C(Y\sqcup \partial Y)\rtimes\Gamma$ $${\rm Ell}(\mathcal A_0,\mathcal A)\cong K_0(C_0(T^*Y^\circ)\rtimes\Gamma)\oplus K_0(C(\partial Y)\rtimes \Gamma).$$ If $\Gamma$ is finitely generated and of polynomial growth, then the elements associated with the second summand do not contribute to the index.