Automating FME Server Docker backups

I haven't looked into the details, but I think you should be mounting a volume for your FME Server database, that way your instance is persisted without having to do backup/restore.

Some details here: https://knowledge.safe.com/questions/57041/docker-swarm-for-fme-server-containers.html

Unfortunately I was unable to find more detailed instruction from Safe on how to do this, I'm sure that's something that could be helpful to a lot of people (paging @GerhardAtSafe :-)

---

I haven't looked into the details, but I think you should be mounting a volume for your FME Server database, that way your instance is persisted without having to do backup/restore.

Some details here: https://knowledge.safe.com/questions/57041/docker-swarm-for-fme-server-containers.html

Unfortunately I was unable to find more detailed instruction from Safe on how to do this, I'm sure that's something that could be helpful to a lot of people (paging @GerhardAtSafe :-)

Thanks @david_r. Agree that I think I need to be mounting a volume and I've done some reading on this. It appears that for Docker for Windows it's not quite as simple as if I were running Docker natively on Linux but it can be done.

 

Also agree that it would be nice to have a bit of a "how to" doc from Safe as I feel that this could be useful to a wider audience, especially with Safe now pushing Docker as the deployment of choice for cloud installs.

 

---

Hi @nic_ran,

so here's my take on this to address some of your questions. In general, the volumes will persist even if the containers are shut down by an unexpected event. The information will only be lost if the volumes are specifically removed by either setting the -v flag with the docker-compose down command or docker volume rm.

 

To make sure that you do not lose any configuration even if the physical server you running FME Server fro Docker on is damaged beyond repair I would definitely recommend making regular FME Server backups. There are a few very easy options to do that with FME Server for Docker:

 

1. Backup to a Resource:

You can run your backup creation workspace on a schedule and write you backups either to an S3 Bucket or a Network Share you have access to.  Here is some documentation.

2. Download the backup via the FME Server REST API:

We have an  FME Server REST API endpoint that allows you to download an FME Server backup. This could be used in various applications including FME Server itself to automate this process.

A more complex solution would be to mount a dedicated folder into your containers via the docker-compose file to have a safe location for your backups. But in the end, this is not much different from adding a network share to your FME Server resources (see solution 1) which makes this the least recommended option for this scenario as modifying the docker-compose file can easily lead to mistakes. One way to do this would be to specify the folder you want to mount for the specific volume. Here is an example of the backups folder:

Before:
- backups:/data/fmeserverdata/resources/backups
After:
- /your_mounted_drive/backups:/data/fmeserverdata/resources/backups

Make sure to change all occurrences of this volume in the docker-compose file accordingly. You can then also remove this reference of the backups folder at volumes list at the end of the docker-compose file:

backups:
      driver: local

I hope this helps!

---

Hi @nic_ran,

so here's my take on this to address some of your questions. In general, the volumes will persist even if the containers are shut down by an unexpected event. The information will only be lost if the volumes are specifically removed by either setting the -v flag with the docker-compose down command or docker volume rm.

 

To make sure that you do not lose any configuration even if the physical server you running FME Server fro Docker on is damaged beyond repair I would definitely recommend making regular FME Server backups. There are a few very easy options to do that with FME Server for Docker:

 

1. Backup to a Resource:

You can run your backup creation workspace on a schedule and write you backups either to an S3 Bucket or a Network Share you have access to.  Here is some documentation.

2. Download the backup via the FME Server REST API:

We have an  FME Server REST API endpoint that allows you to download an FME Server backup. This could be used in various applications including FME Server itself to automate this process.

A more complex solution would be to mount a dedicated folder into your containers via the docker-compose file to have a safe location for your backups. But in the end, this is not much different from adding a network share to your FME Server resources (see solution 1) which makes this the least recommended option for this scenario as modifying the docker-compose file can easily lead to mistakes. One way to do this would be to specify the folder you want to mount for the specific volume. Here is an example of the backups folder:

Before:
- backups:/data/fmeserverdata/resources/backups
After:
- /your_mounted_drive/backups:/data/fmeserverdata/resources/backups

Make sure to change all occurrences of this volume in the docker-compose file accordingly. You can then also remove this reference of the backups folder at volumes list at the end of the docker-compose file:

backups:
      driver: local

I hope this helps!

Hi @GerhardAtSafe,

 

thanks for your detailed response, lots of good information.

 

I am familiar with setting up network shares as a resource in FME Server and before I submitted my original question I had tried defining a Network Share as a resource connection. However, the FME Server instance running under Docker is not able to see my network share and I get a "Path does not exist" error when attempting to create the connection. I am able to see the same UNC path from other computers, hence my assumption that I need to do something special in the Docker compose file to expose network resources.

 

 

Any additional information that you can share specifically with regard to setting up a "Network based resource" connection in FME Server while running under Docker for Windows would be much appreciated.

 

 

As a side note, I have had partial success with your option 3, above, repointing the backups volume to a mounted drive. FME Server has started successfully and the mount worked (I can see another file under the shared resource Backups folder) but an error is encountered when FME Server tries to move the backup file to that folder. Might be a permissions issue and I will investigate further...

 

 

Nic

 

---

Hi @GerhardAtSafe,

 

thanks for your detailed response, lots of good information.

 

I am familiar with setting up network shares as a resource in FME Server and before I submitted my original question I had tried defining a Network Share as a resource connection. However, the FME Server instance running under Docker is not able to see my network share and I get a "Path does not exist" error when attempting to create the connection. I am able to see the same UNC path from other computers, hence my assumption that I need to do something special in the Docker compose file to expose network resources.

 

 

Any additional information that you can share specifically with regard to setting up a "Network based resource" connection in FME Server while running under Docker for Windows would be much appreciated.

 

 

As a side note, I have had partial success with your option 3, above, repointing the backups volume to a mounted drive. FME Server has started successfully and the mount worked (I can see another file under the shared resource Backups folder) but an error is encountered when FME Server tries to move the backup file to that folder. Might be a permissions issue and I will investigate further...

 

 

Nic

 

Hi @nic_ran

 

the 3rd option works for me, so I assume this is related to permissions on your side. I was able to write a backup to my local disk or a network share that was mounted on my machine with FME Server. I use Docker for Mac Version 17.12.0

 

Adding a network share to the FME Server resources in the WebUI is indeed not as easy as it first seemed. The way this works on Linux is that you mount the network share to a folder on the machine FME Server is installed and then you add a FME Server Resource connecting to that new folder. I assumed that we could do the same when we mount the share to a folder in the FME Server Core container. But so far I was not able to mount a share to our containers. I am also not sure if this would be desirable or good practice. I will let you once I know more about this.

 

Ger

---

Hi @nic_ran

 

the 3rd option works for me, so I assume this is related to permissions on your side. I was able to write a backup to my local disk or a network share that was mounted on my machine with FME Server. I use Docker for Mac Version 17.12.0

 

Adding a network share to the FME Server resources in the WebUI is indeed not as easy as it first seemed. The way this works on Linux is that you mount the network share to a folder on the machine FME Server is installed and then you add a FME Server Resource connecting to that new folder. I assumed that we could do the same when we mount the share to a folder in the FME Server Core container. But so far I was not able to mount a share to our containers. I am also not sure if this would be desirable or good practice. I will let you once I know more about this.

 

Ger

Hi @GerhardAtSafe,

 

I've got option 3 working now. I unmounted everything, rebooted the server, restarted the FME Server Docker instance and now it works. Didn't actually change any settings, so not sure why I can now write when I couldn't before but it works!

 

 

As you mentioned in your original answer, editing the compose file is not the ideal solution so I am still interested in hearing if there's a way that this can be achieved with the standard compose file and setting up a "Network based resource" connection through the web UI, specifically for a Docker for Windows deployment.

 

 

Thanks for all of your assistance to date.

 

Nic

 

---