Here is the full quotation (from his concurring opinion in Brown v. Allen, 1953):
“We are not final because we are infallible, but we are infallible only because we are final. This is the inevitable consequence of the rule of law.”
However, the specific phrase you mentioned — “with all its defects, delays, and inconveniences” — comes from a different famous Jackson passage. The full quote is:
“The due process of law, as guaranteed by the Constitution, is the process which has been pursued from time immemorial. It is the law in its regular course of administration through courts of justice. With all its defects, delays, and inconveniences, men have discovered no technique for long preserving free government except that the Executive be under the law.”
— Robert H. Jackson, dissenting opinion in Shaughnessy v. United States ex rel. Mezei (1953)