This phase II trial studies how well fluorouracil, leucovorin calcium, and irinotecan hydrochloride (FOLFIRI) together with panitumumab work in treating patients with colorectal cancer that expresses the RAS and B-Raf proto-oncogene, serine/threonine kinase (BRAF) wild-type genes, has spread from the original site of growth to another part of the body (metastatic), resists the effects of treatment with prior cetuximab (or panitumumab) plus irinotecan hydrochloride-based therapy, and who have failed at least one subsequent non-anti-epidermal growth factor receptor (EGFR) containing treatment regimen. Drugs used in chemotherapy, such as fluorouracil, leucovorin calcium, and irinotecan hydrochloride, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving more than one drug (combination chemotherapy) may kill more tumor cells. Monoclonal antibodies, such as panitumumab, may block tumor growth in different ways by targeting certain cells. Giving FOLFIRI together with panitumumab may be an effective treatment for colorectal cancer.