[FR][OS 6.1.8]Ftp Backup source hidden files

Ftp in this case is Proftpd on ubuntu server 12.04 (from distirbution apt source) .
Files are visible&readable with Filezilla & Winscp & Cyberduck.

There is no FTP spec for showing hidden files. Most UNIX FTP servers can accept a "-a" argument to the FTP LIST command, which they will probably then pass on to a forked "ls" command. But this is certainly not a standard, and I have no idea what this will do to, say, an IIS FTP server.

Hi,
thank you for taking the time.
I'm not sure what your saying here:
1) OS6 FTP backup is looking for hidden files but doesn't get an answer from that (ProFTPd) server although other clients manage it (winscp,filezilla,cyberduck)
2) OS6 FTP backup is not looking for hidden files because there is no standard to garantee the result with raw FTP commands

In order to keep this constructive, I dugg a little bit and LIST reference suggests indeed that we have 2).
I have no way to test a IIS FTP setup but some search suggest that it will not chocke on "-a" and that actually admins need to configure their desired support for "-a" listing requests.

So that leaves us with two points to address :
1) As a Readynas user, I would like to know that my FTP backup tool has either an option or a know limitation as to how it retrieves directory contents. I can't force Netgear to implement my obvious preference (support for -a), but I'd be glad to have other users chime in and explain if they would like an option (or mandatory "-a" usage for all I care... :wink: ) to make FTP Backup directory listing results more comprehensive.
2) As Netgear, I would like to know that my FTP backup tool doesn't make mainstream (at least) FTP Servers to chocke on my commands. Again I can't force Netgear to take the risk but I believe an option to enable/disable "-a" (maybe with a warning about impropable no support).

I believe I kept this constructive and save you the time to build a "case" for your product evolution: :idea: We would greatly benefit from having a by default checked checkbox with "Support hidden files (-a)".

Lastly, would it help if I create a support case or it will just create unecessary triage and overhead?
I made this topic a [FR] feature request...But not sure if correct way....Feel free to correct or suggest correction.

Thanks for any suggestion.

To clarify, the OS6 FTP server on my RN102 does show hidden files to Filezilla (and any other client) in response to the normal MLSD or MLST FTP commands It does this by default. This discussion is specific to the FTP client implementation used in the backup job. Using MLST/MLSD is better than LIST for a variety of reasons, including the support for 8 bit characters in the file lists. Clients should use MLST/MLSD exclusively if the server implements them.

However, the OS6 FTP server does not tell the client that the file is hidden - and in fact it cannot, because there is no standard way to do that in FTP. The convention that ".filename" is hidden is operating system specific, and some operating systems (for instance Windows) don't use that convention.

Also, the newer commands (MLSD and MLIST) don't have options sent in the command line. There is a way to request specific "facts" in the MLST/MLSD response, that is described in RFC 3659 - including modification date, size, owner, group, and permissions. But again, not "hidden".

Finally, the FTP model is that the decision to show or not show hidden files is a server choice, not the client choice. Files are hidden for a reason, and it is up the the file repository to decide if it wants to allow them to be access/seen or not. BTW, this is true for any file or directory, not just hidden ones. The server is not obligated to show any files, or allow access to them. It can choose to filter the list any way it wishes. One reason is practical - there is no way to force server software to show all the files. There are also security aspects - files are generally hidden for a reason. This is why FileZilla makes "force showing hidden files" a server option - and it has no client option.

So
(1) there is in fact no good way to tell the FTP server to show hidden files. The old LIST command with -a might work, but might return an error or an empty list - the behavior is not defined. The new commands that support 8 bit character codes don't support that option at all.
(2) Some servers choose to show hidden files, some don't, and for some (FileZilla for instance) it is an option.
(3) if you do receive (or send) hidden files from the backup client, then when you back them up they might not stay hidden. It depends on the destination OS and file system.

My thought is that the correct behavior for the FTP backup client is to read or write all the files it finds, with the exception of "." and ".." and other known "special" file names that cannot be read or written. So if hidden files are shown in the source, the client will try to read them. And if FTP is the destination protocol, then the client will try to write them to the destination. I don't off hand see why this should be optional behavior, since the FTP client can't identify hidden files anyway. But if the FTP client is filtering out files that start with "." for some reason, then an option to defeat that filtering would be a good feature.

Adding a "LIST -a" option could create support headaches for Netgear if it is used - since there are almost certainly servers that won't accept that "-a", and will misbehave if you send it to them. And if the FTP server supports MLST/MLSD, they should be used instead, and they don't certainly don't allow "-a". So I'm thinking it isn't a good idea.

Though of course Netgear will do as it wishes - this is just what I'd do if I were them.

FTP wouldn't be the protocol I would recommend for backups anyway. I would consider Rsync which is a native Linux protocol designed for backups, especially backups over WAN.

If you are using a Windows you could install DeltaCopy and update Rsync etc. using the latest versions from Cygwin

The OP is wanting to back up a linux server. I agree rsync would be better.

That said, being able to transfer data over FTP via a backup job is handy, and has other uses than strictly backup.

Indeed, this is not about the best choice and this has nothing to do with backup (which is only a logical wording Netgear chose to expose their FTP Client).
It is about understanding and enhancing (maybe) that FTP Client behaviour so that it is consistent with the server exposure of hidden files.
FTP is very very very handy on simpler machines.

Well, if Netgear wanted me to fix it, I'd

(a) make sure MLST/MLSD are being used whenever they are advertised via the FTP FEAT command (may be true already)
(b) remove any hard-wired filters for .*
(c) make sure there is a hardwired filter for . and ..
(d) implement the exclude file feature in frontview backup for FTP, so .* could be manually excluded.

I wouldn't mess around with LIST -a

That should match the behavior of the FileZilla client, which based on your earlier posts is what you are asking for. Anyway, I think those adjustments would increase the utility of the FTP backup job.

I think you got it cornered here!
Now let's wait for Jedi-mdgm-ification....