It is interesting that the Copy activity in your pipeline behaves differently depending on how it is being configured. If you do not specify the filename in the sink folder, it will keep appending the parquet file with different file names. If the filename in the sink folder is specified, it will overwrite the existing parquet file.
As an example, the asset master folder contains 2 parquet files with different timestamps. In this case, the filename has not been specified.
Files in folder abfs://…/asset_master/:
Number of rows: , 2556
Number of rows: , 2567
For the case when the filename in the sink is specified, ADF will overwrite the existing parquet file.
@concat(dataset().FolderName, ‘.parquet’)
First ingestion:
Second ingestion:
Leave a comment