APS (aka. PDW) Destination Adaptor "“Item has already been added. Key in dictionary: '' Key being added: ''”"

In couple of packages in our project got error “Item has already been added. Key in dictionary: '<number>' Key being added: '<number>'”. After observing properly issue was resolved. Here is detailed explanation of the issue.

In Data Flow Task (DFT) there source lookup and destination adaptor. From source one date was selected; let’s call it ‘FromDate’.  In Lookup one date of fetched and output name of date was also ‘FromDate’. So now the downstream pipeline was having two date columns with same name.

All other defaults transformation of SSIS will resolve this conflict by appending the Lookup to the column. But the PDW destination adaptor was unable to do this. Hence it was giving error as “Item has already been added. Key in dictionary: '<number>' Key being added: '<number>'”

Example of columns with same name in pipeline

Example of components solving the same column name by appending component name

Solution: Name the columns uniquely in the pipeline. Since the PDW destination adaptor is not capable of handling same name column as other components do.

Keyword: APS, PDW, Destinaton, Microsoft Parallel Data Warehouse, Microsoft Analytics Platform