Season the fish with salt and pepper. In a very large skillet, warm 1/4 cup of the olive oil over high heat. Add the fish and cook, without moving, until browned on one side, about 2 minutes. (Cook in batches if necessary to avoid overcrowding the pan.) Transfer the fish to a platter and set aside.
In the same skillet, warm the remaining 1/4 cup olive oil over medium-high heat. Add the capers and fry until they start to brown, 1 to 2 minutes. Add the olives, garlic, fennel seeds and chili flakes and cook until everything is toasty and fragrant, about 2 minutes. Add 2 tablespoons of water to the pan. Remove from the heat.
In a large pot of well-salted boiling water, cook the pasta according to the package instructions until 2 minutes shy of al dente; drain.
Toss the paccheri into the skillet with the caper mixture, along with the swordfish. Cook over medium heat until the pasta is al dente, 1 to 2 minutes, adding more water if the sauce seems dry. Stir in the parsley and mint and season with salt, pepper and lemon juice.
Divide the pasta among four individual serving plates or bowls and finish each with drizzle of olive oil.