I know how to copy files between servers using rsync.
But I was looking for an option where I donot want to copy entire content file on another server.
Example:- I want to copy /var/log/messages filtered by todays date on remote server in todays date folder and tomorrows content in tomorrows date folder and so on. Though I can achieve the same by creating temporary files in local node before transferring, but I am looking for a solution where filtered data can be transferred to remote server without creating any local temporary file
About linux : copy-filtered-content-of-file-to-remote-server-via-rsync
Question Detail
Question Answer
where filtered data can be transferred to remote server without creating any local temporary file
Just the same way you do locally, just cat the output to the remote. Along:
cat /var/log/messages | awk 'some filtering here' | ssh server 'cat > /the/destination'
Repeat the process for each file.
The simpler is just the same as creating temporary files you want to transfer – just create them at the destination! Mount sshfs the remote and create them inside the mountpoint.
I am looking forward for a solution which works with rsync
It’s easy to say: create a FUSE filesystem that applies mentioned filtering on top of existing “bottom” paths transparently. So that reading from a file will result in reading the filtered data only. This sounds like a rather extensive project involving a lot of time. Then use rsync of top of such FUSE filesystem. The filtering will then be done transparently by the code implementing the filesystem.
Another idea: overall, as of now, rsync does not support any kind of “content filtering/parsing” to the transferred files. rsync is overall a network communication protocol. Take rsync sources and implement such functionality inside the rsync server – before transferring each file, apply a function that transforms the file content, i.e. does the filtering that you want.