We evaluate the performance of two popular systemic risk measures, CoVaR and SRISK, during eight financial panics in the era before FDIC insurance. Bank stock price and balance sheet data were not readily available for this time period. We rectify this shortcoming by constructing a novel dataset for the New York banking system before 1933. Our evaluation exercise focuses on two challenges: the ranking of systemically important financial institutions (SIFIs) and financial crisis prediction. We find that CoVaR and SRISK meet the SIFI ranking challenge, i.e. help identifying systemic institutions in periods of distress beyond what is explained by standard risk measures up to six months prior to panics. In contrast, aggregate CoVaR and SRISK are only somewhat effective at predicting financial crises.