Thirty years after the ‘Bilbao Effect,’ museums are shunning exuberant form-making in favor of thoughtful programming that engages new audiences.