NAS Backup - Raise Event for Succeeded / Failed / Timed Out

[daviftorres]

The required feature described as a wish

As an operator, I’d like to improve event visibility for NAS backup jobs, especially when they fail.

On Sep 17th, scheduled backups started but never completed.

The task metadata only showed Status = BackingUp, as if it were still running, with no further updates.

No backup-related event was raised for that date (with no filters applied).

In the Filter Types, I couldn’t find any category for NAS Backup, not even under Storage.

This makes it difficult to detect and track incomplete or failed backups.

I would like to monitor events using the Webhook feature but they need to be raised (exist).

[daviftorres]

Looking at the code (https://github.com/apache/cloudstack/blob/f63118c011ecd90a81adee5fd043a966b4822c47/plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java) I see the error handlers:

• Line 223 sets status to Failed when connection fails,
• Line 228 sets status to Failed when times out,
• Line 251 sets status to Failed when fails for any other reason.

Note that in line 247 sets status to Error when the cleanup fails, and that is the only time it updates and leave the backup with a failing statatus behind. Al the other scenarios it removed right after setting status.

Why not keeping the failed tentative in the logs?
Can we generate an events / alerts for those failed conditions? (see example below)

[abh1sar]

Looking at the code (https://github.com/apache/cloudstack/blob/f63118c011ecd90a81adee5fd043a966b4822c47/plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java) I see the error handlers:

• Line 223 sets status to Failed when connection fails,
• Line 228 sets status to Failed when times out,
• Line 251 sets status to Failed when fails for any other reason.

Note that in line 247 sets status to Error when the cleanup fails, and that is the only time it updates and leave the backup with a failing statatus behind. Al the other scenarios it removed right after setting status.

Hi @daviftorres ,
The idea with setting error on line 247 and not deleting the Backup entry is that in this case after some error, the backend wasn't able to cleanup the backup files in the storage. So, the user has to cleanup those files manually and then delete the backup.

In case of other errors, there are no backup files created on the storage, so the Backup can be deleted by the system after returning an appropriate error. If we leave it, that is just manual overhead for the user to delete.
The User will see the error in the UI and it is being logged as well. I believe volume snapshots have the same behaviour.

Setting the status as Failed just before removing doesn't solve much, but I guess it is still useful for forensics when someone looking at the backups table in the db.

Usually, CloudStack doesn't create Alerts for failure events, they are mostly for monitoring capacities such as storage space etc.
But Event for backup failures at least for scheduled backups could be a good idea. I don't see such events for Scheduled snapshots as well, so may be there could be a generic solution for such events.

Hope that clarifies it. Happy to discuss further.

About the Issue of Backups being stuck in the BackingUp state, which version are you running?
There was a bug in the nasbackup.sh script, due to which, if there was some connection error or mount error, Backup job could be stuck indefinitely or for a very long time. I fixed it in 4.21. I haven't seen backup jobs stuck in my testing so far in 4.21 and 4.22

79f83db

[daviftorres]

Hey @abh1sar , thank you for your reply. It was very enlightening.

You are correct, the issue is on ACS 4.20 and I saw the fix you provided.

I am satisfied with the outcomes to this issue report and I will happily close it.

Keep up with the good work!

[abh1sar]

Usually, CloudStack doesn't create Alerts for failure events, they are mostly for monitoring capacities such as storage space etc. But Event for backup failures at least for scheduled backups could be a good idea. I don't see such events for Scheduled snapshots as well, so may be there could be a generic solution for such events.

Reopening this issue to add tracking for failure events, specially for scheduled jobs.

[daviftorres]

Hey @abh1sar ,

Thank you for seeing the value of the points I raised. As an example, the following image shows the events when backups are created and have the state Started. There is no evidence if they succeeded or failed.

Note: only the top one succeeded. The three below that failed.

Regards,

[daviftorres]

@abh1sar

[abh1sar]

Note: only the top one succeeded. The three below that failed.

Regards,

Yes, this needs to be fixed. A ‘Backup create completed’ event should at least be thrown..