We evaluate the performance of two popular systemic risk measures, CoVaR and SRISK, during eight financial panics in the era before FDIC insurance. Bank stock price and balance sheet data were not readily available for this time period. We rectify this shortcoming by constructing a novel dataset for the New York banking system before 1933. Our evaluation exercise focuses on two challenges: the ranking of systemically important financial institutions (SIFIs) and financial crisis prediction.