• Uncategorized

About linux : copy-filtered-content-of-file-to-remote-server-via-rsync

Question Detail

I know how to copy files between servers using rsync.
But I was looking for an option where I donot want to copy entire content file on another server.
Example:- I want to copy /var/log/messages filtered by todays date on remote server in todays date folder and tomorrows content in tomorrows date folder and so on. Though I can achieve the same by creating temporary files in local node before transferring, but I am looking for a solution where filtered data can be transferred to remote server without creating any local temporary file

Question Answer

where filtered data can be transferred to remote server without creating any local temporary file

Just the same way you do locally, just cat the output to the remote. Along:

cat /var/log/messages | awk 'some filtering here' | ssh server 'cat > /the/destination'

Repeat the process for each file.

The simpler is just the same as creating temporary files you want to transfer – just create them at the destination! Mount sshfs the remote and create them inside the mountpoint.


I am looking forward for a solution which works with rsync

It’s easy to say: create a FUSE filesystem that applies mentioned filtering on top of existing “bottom” paths transparently. So that reading from a file will result in reading the filtered data only. This sounds like a rather extensive project involving a lot of time. Then use rsync of top of such FUSE filesystem. The filtering will then be done transparently by the code implementing the filesystem.

Another idea: overall, as of now, rsync does not support any kind of “content filtering/parsing” to the transferred files. rsync is overall a network communication protocol. Take rsync sources and implement such functionality inside the rsync server – before transferring each file, apply a function that transforms the file content, i.e. does the filtering that you want.

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.