Advice on ETL pipeline for large and varied files

Hello to the amazing community. It is time for me to build an ETL pipeline. Problem facts - Data arrives in S3 in various file formats. - Files vary in size, structure wildly. So a lot of file specific rules, mappings, transformations are needed to be maintained. Also post processing rules vary per file. - Files are large, frequent and unreliable(arrive sporadically even though they're supposed to have a set schedule and that's also part of the problem - unreliability of publishers) Ultimate goal - to feed a bunch of different reporting and other features including data search and download. Th