This phase II trial studies how well umbralisib and pembrolizumab work in treating patients with classical Hodgkin lymphoma that has come back (relapsed) or does not respond to treatment (refractory). Umbralisib may stop the growth of cancer cells by blocking some of the enzymes needed for cell growth. Immunotherapy with monoclonal antibodies, such as pembrolizumab, may help the body's immune system attack the cancer, and may interfere with the ability of cancer cells to grow and spread. Giving umbralisib and pembrolizumab may work better in treating classical Hodgkin lymphoma.