The group flew to Rome to watch the cardinal testify before a Royal Commission after he was excused from returning home due to ill health.
They said they were dissatisfied with the testimony of Cardinal Pell, who is Vatican treasurer.
Survivors said they requested via email to meet the Pope last week.
"We are flying back to Australia on Friday and we'd like to think we can get a meeting," survivor Philip Nagle said.
"[Pell] is giving us nothing. He doesn't care. He's turned his back on us. We don't want to meet with George at all."
Cardinal Pell was questioned over his knowledge of the priest Peter Searson on day three of his testimony.
Searson died in 2009 without facing charges, despite a litany of complaints against him ranging from sexual abuse of children to animal cruelty.
Cardinal Pell told the inquiry he was not adequately briefed by the Catholic Education Office, who told him about the allegations against Searson in a "non-specific way".
The cardinal said that when the Catholic Education Office told him the matter was resolved, he took their word for it.
Cardinal Pell said Searson was "one of the most unpleasant priests that [he had] met".
