We have couple of CSV files on Azure Container Folder and accessing those files with Polybase External Table.
All the files has the same structure, data of each file can be queried when accessed separately.
CREATE EXTERNAL TABLE [dbo].[dbs_ACC_DATA]
(
[Account Number] [varchar](50) NULL,[Client ID] [varchar](100) NULL,[Active?] [varchar] (10) NULL,[Individual or Entity Account] [varchar] (10) NULL)
WITH (DATA_SOURCE = [AzureDS],LOCATION = N'/client_files_dbs/file_1.csv',FILE_FORMAT =[az_storage_files],REJECT_TYPE = VALUE,REJECT_VALUE = 1)
SELECT * FROM dbs_ACC_DATA
We are running into the following issues when accessing all the files in the folder
CREATE EXTERNAL TABLE [dbo].[dbs_ACC_DATA]
(
[Account Number] [varchar](50) NULL,[Client ID] [varchar](100) NULL,[Active?] [varchar] (10) NULL,[Individual or Entity Account] [varchar] (10) NULL)
WITH (DATA_SOURCE = [AzureDS],LOCATION = N'/client_files_dbs/',FILE_FORMAT =[az_storage_files],REJECT_TYPE = VALUE,REJECT_VALUE = 1)
Error :
Cannot execute the query "Remote Query" against OLE DB provider "MSOLEDBSQL" for linked server "(null)". 107090;Query aborted-- the maximum reject threshold (1 rows) was reached while reading from an external source: 2 rows
rejected out of total 2 rows processed.
Error log from DWEngine_server log file.
CREATE EXTERNAL TABLE [dbo].[dbs_ACC_DATA]
(
[Account Number] [varchar](50) NULL, [Client ID] [varchar](100) NULL, [Active?] [varchar] (10) NULL, [Individual or Entity Account] [varchar] (10) NULL, [Single or Joint Account] [varchar] (10) NULL,
[Number of Account Holders] [varchar] (10) NULL, [Entity Name] [varchar](300) NULL, [DBA/Disregarded Entity Name] [varchar](100) NULL, [Country of Organization/Incorporation] [varchar](2) NULL,
[First Name] [varchar](300) NULL, [Middle Name] [varchar](300) NULL, [Last Name] [varchar](300) NULL, [Citizenship Country] [varchar](2) NULL, [Country of Birth] [varchar](2) NULL, [Date of Birth] [varchar](100) NULL,
[PR Address - Street 1] [varchar](150) NULL, [PR Address - Street 2] [varchar](150) NULL, [PR Address - City] [varchar](100) NULL, [PR Address - Postal Code] [varchar](50) NULL, [PR Address - State] [varchar](100) NULL, [PR Address - Country] [varchar](2) NULL,
[Mailing Address - Street 1] [varchar](150) NULL, [Mailing Address - Street 2] [varchar](150) NULL, [Mailing Address - City] [varchar](100) NULL, [Mailing Address - Postal Code] [varchar](50) NULL, [Mailing Address - State] [varchar](100) NULL,
[Mailing Address - Country] [varchar](2) NULL, [Additional Mailing Address Countries] [nvarchar](100) NULL, [Phone Number] [nvarchar](30) NULL, [Standing Instructions Country] [varchar](2) NULL,
[Account Opening Date] [datetime] NULL, [Classification of a U.S. Person] [int] NULL, [Foreign TIN Type] [nvarchar](6) NULL,
[U.S. TIN] [varchar](9) NULL, [U.S. TIN Type] [nvarchar](6) NULL, [Registration Code] [nvarchar](100) NULL, [Email Address] [varchar](200) NULL, [Title] [varchar](100) NULL,
[Town of Birth] [varchar](50) NULL, [Reporting FI Jurisdiction] [varchar](2) NULL, [Tax Residence] [nvarchar](2) NULL, [POA Country] [varchar](2) NULL,
[Hold Mail Country] [varchar](2) NULL, [CIC Code] [int] NULL, [FILER1] [varchar](100) NULL, [FILER2] [varchar](100) NULL, [FILER3] [varchar](100) NULL, [FILER4] [varchar](100) NULL, [FILER5] [varchar](100) NULL,
[FILER6] [varchar](100) NULL, [FILER7] [varchar](100) NULL, [FILER8] [varchar](100) NULL, [FILER9] [varchar](100) NULL, [FILER10] [varchar](100) NULL
)
WITH (DATA_SOURCE = [AzureDS],LOCATION = N'/client_files_dbs/',FILE_FORMAT =[az_storage_files],REJECT_TYPE = VALUE,REJECT_VALUE = 1)
SID333:QID507 [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:05 AM [Thread:3652] [ServerInterface:InformationEvent] (Info, Normal): Starting processor SqlFrontEndWorkProcessor. [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:05 AM [Thread:3652] [ExternalHadoopBridge:InformationEvent] (Info, Normal): Initializing bridge... [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:05 AM [Thread:3652] [JavaBridge:InformationEvent] (Info, Normal): LoadOrAttachJVM: Getting created JVMs if any... [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:05 AM [Thread:3652] [JavaNativeInterface:InformationEvent] (Info, Normal): Attaching JVM to current thread... [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:05 AM [Thread:3652] [ExternalHadoopBridge:InformationEvent] (Info, Normal): Attached to JVM... [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:05 AM [Thread:3652] [HdfsBridgeFileAccess:InformationEvent] (Info, Normal): GetFileMetadata... [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:05 AM [Thread:3652] [JavaBridge:InformationEvent] (Info, Normal): LoadOrAttachJVM: Getting created JVMs if any... [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:05 AM [Thread:3652] [JavaNativeInterface:InformationEvent] (Info, Normal): Attaching JVM to current thread... [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:05 AM [Thread:3652] [ExternalHadoopBridge:InformationEvent] (Info, Normal): Attached to JVM... [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:05 AM [Thread:3652] [HdfsBridgeJniWrapper:InformationEvent] (Info, Normal): Acquired JvmCreationLock... [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:10 AM [Thread:3652] [HdfsBridgeFileAccess:InformationEvent] (Info, Normal): Getting file info for each file in a directory containing total 4 files [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:10 AM [Thread:3652] [HdfsBridgeFileAccess:InformationEvent] (Info, Normal): File info obtained for each file. [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:10 AM [Thread:3652] [ServerInterface:InformationEvent] (Info, Normal): Query SID333:QID507 completed. [Session.SessionId:SID333][Session.IsTransactional:False][Query.QueryId:QID507]
8/23/2019 12:28:10 AM [Thread:4960] [EngineInstrumentation:EngineStopSessionBeginEvent] (Info, Low): Stopping session, in reset connection: False. [Session.SessionId:SID333][Session.IsTransactional:False]
8/23/2019 12:28:10 AM [Thread:4960] [EngineInstrumentation:TdsRemoveConnectionEvent] (Info, Low): Stopping TDS connection: 2609209822992
8/23/2019 12:28:10 AM [Thread:4556] [TdsRequestProcessor:InformationEvent] (Info, Normal): CancelBatch request started [Session.SessionId:SID333]
8/23/2019 12:28:10 AM [Thread:4556] [EngineInstrumentation:TdsRemoveConnectionEvent] (Info, Low): Stopping TDS connection: 2609209824864
8/23/2019 12:28:10 AM [Thread:4556] [EngineInstrumentation:EngineStopSessionBeginEvent] (Info, Low): Stopping session, in reset connection: False. [Session.SessionId:SID333][Session.IsTransactional:False]
8/23/2019 12:28:12 AM [Thread:4556] [PdwTdsNativeToManagedInterop:InformationEvent] (Info, Normal): Incoming TDS connection, Client TDS version: 7 (TDS74).
8/23/2019 12:28:12 AM [Thread:4556] [EngineInstrumentation:TdsAddConnectionEvent] (Info, Low): Starting TDS connection: 2609209822992 Client: 192.168.10.49:51536, isFromDataSecurityProxy: False, isVnetAddress: False, clientAddressSafe: 192.168.10.49:51536
8/23/2019 12:28:12 AM [Thread:4556] [DataClassificationConfig:InformationEvent] (Info, Normal): Returning FeatureSwitch DataClassificationCoreEnabled status: False [Session.SessionId:SID334][Session.IsTransactional:False]
8/23/2019 12:28:12 AM [Thread:4556] [EngineInstrumentation:ServerStartSessionEvent] (Info, Low): Started new session, in reset connection: False. [Session.SessionId:SID334][Session.IsTransactional:False]
8/23/2019 12:28:12 AM [Thread:4556] [TdsRequestProcessor:InformationEvent] (Info, Normal): Processing login, authentication scheme: "Ntlm", client application name: "87E0B5F8-7B5A-776E-2D52-AA5CB888E28D"
8/23/2019 12:28:12 AM [Thread:4556] [PdwTdsNativeToManagedInterop:InformationEvent] (Info, Normal): Data Classification TDS Extension Requested=False, FS Enabled=False, Enabled=False. [Session.SessionId:SID334][Session.IsTransactional:True][Query.QueryId:QID508]
8/23/2019 12:28:12 AM [Thread:1668] [EngineInstrumentation:TdsAddConnectionEvent] (Info, Low): Starting TDS connection: 2609209824864 Client: , isFromDataSecurityProxy: False, isVnetAddress: False, clientAddressSafe:
8/23/2019 12:28:12 AM [Thread:4556] [EngineInstrumentation:EngineStopSessionBeginEvent] (Info, Low): Stopping session, in reset connection: False. [Session.SessionId:SID334][Session.IsTransactional:False]
8/23/2019 12:28:12 AM [Thread:4556] [EngineInstrumentation:TdsRemoveConnectionEvent] (Info, Low): Stopping TDS connection: 2609209824864
8/23/2019 12:28:12 AM [Thread:4556] [TdsRequestProcessor:InformationEvent] (Info, Normal): CancelBatch request started [Session.SessionId:SID334]
8/23/2019 12:28:12 AM [Thread:4636] [EngineInstrumentation:EngineCancelQueryBeginEvent] (Info, Low): Cancel requested. [Session.SessionId:SID334][Session.IsTransactional:False]
8/23/2019 12:28:12 AM [Thread:4556] [EngineInstrumentation:TdsRemoveConnectionEvent] (Info, Low): Stopping TDS connection: 2609209822992
8/23/2019 12:28:12 AM [Thread:4556] [EngineInstrumentation:EngineStopSessionBeginEvent] (Info, Low): Stopping session, in reset connection: False. [Session.SessionId:SID334][Session.IsTransactional:False]
8/23/2019 12:28:12 AM [Thread:4556] [PdwTdsNativeToManagedInterop:InformationEvent] (Info, Normal): Incoming TDS connection, Client TDS version: 7 (TDS74).
8/23/2019 12:28:12 AM [Thread:4556] [EngineInstrumentation:TdsAddConnectionEvent] (Info, Low): Starting TDS connection: 2609209822992 Client: 192.168.10.49:51537, isFromDataSecurityProxy: False, isVnetAddress: False, clientAddressSafe: 192.168.10.49:51537
8/23/2019 12:28:12 AM [Thread:4556] [DataClassificationConfig:InformationEvent] (Info, Normal): Returning FeatureSwitch DataClassificationCoreEnabled status: False [Session.SessionId:SID335][Session.IsTransactional:False]
8/23/2019 12:28:12 AM [Thread:4556] [EngineInstrumentation:ServerStartSessionEvent] (Info, Low): Started new session, in reset connection: False. [Session.SessionId:SID335][Session.IsTransactional:False]
8/23/2019 12:28:12 AM [Thread:4556] [TdsRequestProcessor:InformationEvent] (Info, Normal): Processing login, authentication scheme: "Ntlm", client application name: "87E0B5F8-7B5A-776E-2D52-AA5CB888E28D"
8/23/2019 12:28:12 AM [Thread:4556] [PdwTdsNativeToManagedInterop:InformationEvent] (Info, Normal): Data Classification TDS Extension Requested=False, FS Enabled=False, Enabled=False. [Session.SessionId:SID335][Session.IsTransactional:True][Query.QueryId:QID509]
8/23/2019 12:28:12 AM [Thread:1668] [EngineInstrumentation:TdsAddConnectionEvent] (Info, Low): Starting TDS connection: 2609209824864 Client: , isFromDataSecurityProxy: False, isVnetAddress: False, clientAddressSafe:
8/23/2019 12:28:12 AM [Thread:4556] [ServerInterface:InformationEvent] (Info, Normal): Incoming Query: SID335:QID510 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:12 AM [Thread:3652] [ServerInterface:InformationEvent] (Info, Normal): Starting processor ExecuteMemoProcessor. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [OptimizedStatement:InformationEvent] (Info, Normal): Memo compilation time: 142.5795 ms [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [OptimizedStatement:InformationEvent] (Info, Normal): Sql Server Optimization Clock: 0 s, CPU: 0.002 s. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [OptimizedStatement:InformationEvent] (Info, Normal): SQL Server XML generation Clock : 0.011 s, CPU: 0.01 s [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [ExternalHadoopBridge:InformationEvent] (Info, Normal): Initializing bridge... [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [JavaBridge:InformationEvent] (Info, Normal): LoadOrAttachJVM: Getting created JVMs if any... [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [JavaNativeInterface:InformationEvent] (Info, Normal): Attaching JVM to current thread... [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [ExternalHadoopBridge:InformationEvent] (Info, Normal): Attached to JVM... [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [HdfsBridgeFileAccess:InformationEvent] (Info, Normal): GetFileMetadata... [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [JavaBridge:InformationEvent] (Info, Normal): LoadOrAttachJVM: Getting created JVMs if any... [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [JavaNativeInterface:InformationEvent] (Info, Normal): Attaching JVM to current thread... [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [ExternalHadoopBridge:InformationEvent] (Info, Normal): Attached to JVM... [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:13 AM [Thread:3652] [HdfsBridgeJniWrapper:InformationEvent] (Info, Normal): Acquired JvmCreationLock... [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [HdfsBridgeFileAccess:InformationEvent] (Info, Normal): Getting file info for each file in a directory containing total 4 files [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [HdfsBridgeFileAccess:InformationEvent] (Info, Normal): File info obtained for each file. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [AppConfigPropertyResolver:InformationEvent] (Info, Normal): Configuration property [ReaderHashCost] has been initialized. Source provider: LocalFileSettingsProvider. Value: 0.00021. Default: 0.00021 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [AppConfigPropertyResolver:InformationEvent] (Info, Normal): Configuration property [WriterCost] has been initialized. Source provider: LocalFileSettingsProvider. Value: 2.4E-05. Default: 2.4E-05 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [AppConfigPropertyResolver:InformationEvent] (Info, Normal): Configuration property [InsBulkCpyCost] has been initialized. Source provider: LocalFileSettingsProvider. Value: 0.00024. Default: 0.00024 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [AppConfigPropertyResolver:InformationEvent] (Info, Normal): Configuration property [NetworkCost] has been initialized. Source provider: LocalFileSettingsProvider. Value: 1E-09. Default: 1E-09 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [AppConfigPropertyResolver:InformationEvent] (Info, Normal): Configuration property [ExternalReadCost] has been initialized. Source provider: LocalFileSettingsProvider. Value: 4.8E-05. Default: 4.8E-05 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [AppConfigPropertyResolver:InformationEvent] (Info, Normal): Configuration property [ExternalHadoopWriteCost] has been initialized. Source provider: LocalFileSettingsProvider. Value: 0.00024. Default: 0.00024 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [AppConfigPropertyResolver:InformationEvent] (Info, Normal): Configuration property [StreamWriteCost] has been initialized. Source provider: LocalFileSettingsProvider. Value: 0.00024. Default: 0.00024 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [LimitNumberOfScaOpConst:InformationEvent] (Info, Normal): The number of literals in the query: 0. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [AppConfigPropertyResolver:InformationEvent] (Info, Normal): Configuration property [DwServiceSoftLimitSchemaEnabled] has been initialized. Source provider: Microsoft.SqlServer.DataWarehouse.Configuration.FeatureSwitchSettingsProvider. Value: False. Default: False [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [AppConfigPropertyResolver:InformationEvent] (Info, Normal): Configuration property [DwServiceSoftLimitQueryOptEnabled] has been initialized. Source provider: Microsoft.SqlServer.DataWarehouse.Configuration.FeatureSwitchSettingsProvider. Value: False. Default: False [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:14 AM [Thread:3652] [OptimizedStatement:InformationEvent] (Info, Normal): Distributed QO time: 1541.9893 ms [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [ExternalFile:InformationEvent] (Info, Normal): For file format DelimitedText, max # of external readers per node has been set to 8 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSPlanBuilder:InformationEvent] (Info, Normal): Table has clustered indexes, or this is a streaming plan. Parallel writers will be disabled during this operation. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [ExternalFile:InformationEvent] (Info, Normal): For file format DelimitedText, max # of external readers per node has been set to 8 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [HadoopFile:InformationEvent] (Info, Normal): Total # of files: 4, Total file splits: 4 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [ExternalFile:InformationEvent] (Info, Normal): For file format DelimitedText, max # of external readers per node has been set to 8 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSPlanBuilder:InformationEvent] (Info, Normal): [BuildExternalMoveReaders] Total 4 split lists for 1 compute nodes each can use up-to 8 external readers. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSPlanBuilder:InformationEvent] (Info, Normal): [BuildExternalMoveReaders]
Node [SQLEXT2016M01:1433]; # of readers used: 4; Reader 0, # of splits: 1; Reader 1, # of splits: 1; Reader 2, # of splits: 1; Reader 3, # of splits: 1;
[Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSPlanBuilder:InformationEvent] (Info, Normal): Table has clustered indexes, or this is a streaming plan. Parallel writers will be disabled during this operation. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSPlanBuilder:InformationEvent] (Info, Normal): Table has clustered indexes, or this is a streaming plan. Parallel writers will be disabled during this operation. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSPlanBuilder:InformationEvent] (Info, Normal): Table has clustered indexes, or this is a streaming plan. Parallel writers will be disabled during this operation. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSPlanBuilder:InformationEvent] (Info, Normal): Table has clustered indexes, or this is a streaming plan. Parallel writers will be disabled during this operation. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSPlanBuilder:InformationEvent] (Info, Normal): Table has clustered indexes, or this is a streaming plan. Parallel writers will be disabled during this operation. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSPlanBuilder:InformationEvent] (Info, Normal): Table has clustered indexes, or this is a streaming plan. Parallel writers will be disabled during this operation. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSPlanBuilder:InformationEvent] (Info, Normal): Table has clustered indexes, or this is a streaming plan. Parallel writers will be disabled during this operation. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSPlanBuilder:InformationEvent] (Info, Normal): Table has clustered indexes, or this is a streaming plan. Parallel writers will be disabled during this operation. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [AppConfigPropertyResolver:InformationEvent] (Info, Normal): Configuration property [DwDmsHighResourcePlanPercentage] has been initialized. Source provider: Microsoft.SqlServer.DataWarehouse.Configuration.XdbSettingsProvider. Value: -1. Default: -1 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DmsConcurrency:InformationEvent] (Info, Normal): DwDmsHighResourcePlanPercentage is loaded: [-1]. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DmsConcurrency:InformationEvent] (Info, Normal): Check if this instance is DwService Gen1 : [False]. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSManager:InformationEvent] (Info, Normal): ExecuteCommand Query:SID335 Plan:a55cf35f-387a-46de-bb2a-4bf6791c925b [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:16 AM [Thread:3652] [DMSManager:InformationEvent] (Info, Normal): DMS Manager starting query: SID335, plan: a55cf35f-387a-46de-bb2a-4bf6791c925b. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:23 AM [Thread:4328] [DMSManager:InformationEvent] (Info, Normal): ExecuteCommand Query:SID335 Plan:61ddf166-0d34-4537-b799-09b2e7f65aa8 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:23 AM [Thread:4328] [DMSManager:InformationEvent] (Info, Normal): DMS Manager starting query: SID335, plan: 61ddf166-0d34-4537-b799-09b2e7f65aa8. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:23 AM [Thread:4296] [DMSManager:InformationEvent] (Info, Normal): Handle de-registered PBQTable{0F05B8EF-B910-4F5B-976E-40AF27BF85DA}
8/23/2019 12:28:27 AM [Thread:4296] [CommandWorker:InformationEvent] (Info, Normal): QueryId QID510 PlanId a55cf35f-387a-46de-bb2a-4bf6791c925b StepId 3: Canceling plan due to reject threshold reached.
8/23/2019 12:28:27 AM [Thread:3812] [CommandWorker:InformationEvent] (Info, Normal): Cancel triggered for data movement
8/23/2019 12:28:28 AM [Thread:360] [CommandWorker:ErrorEvent] (Error, High): Plan failed with DMS CommandWorker Exception:
Microsoft.SqlServer.DataWarehouse.Common.ErrorHandling.MppSqlException[107090:1]: 107090;Query aborted-- the maximum reject threshold (1 rows) was reached while reading from an external source: 2 rows rejected out of total 2 rows processed. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:28 AM [Thread:3652] [AbstractDataMovementExecutable`1:ErrorEvent] (Error, High): Data movement error detected- Code: 0
Microsoft.SqlServer.DataWarehouse.Common.ErrorHandling.MppSqlException[107090:1]: 107090;Query aborted-- the maximum reject threshold (1 rows) was reached while reading from an external source: 2 rows rejected out of total 2 rows processed. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:28 AM [Thread:3652] [AbstractDataMovementExecutable`1:InformationEvent] (Info, Normal): DMS Manager finishing query: SID335, plan: a55cf35f-387a-46de-bb2a-4bf6791c925b, queryId: QID510 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:28 AM [Thread:3652] [EngineInstrumentation:QueryStepExecuteErrorEvent] (Error, Critical):
Microsoft.SqlServer.DataWarehouse.Common.ErrorHandling.MppSqlException[107090:1]: 107090;Query aborted-- the maximum reject threshold (1 rows) was reached while reading from an external source: 2 rows rejected out of total 2 rows processed. ---> Microsoft.SqlServer.DataWarehouse.Common.ErrorHandling.MppSqlException[107090:1]: 107090;Query aborted-- the maximum reject threshold (1 rows) was reached while reading from an external source: 2 rows rejected out of total 2 rows processed.
--- End of inner exception stack trace ---
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.AbstractDataMovementExecutable`1.HandleError()
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.AbstractDataMovementExecutable`1.OnExecute(ISessionContext Session)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.AbstractExecutable`1.Execute(ISessionContext session)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.ExecutableProcessor.<>c__DisplayClass40_0.<ExecuteExecutable>b__0(IList`1 propagationTokens)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.ExecutableProcessor.ExecuteActionInDistributedTransactions(Int32 transactionsCount, Action`1 action, IList`1 propagationTokens)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.ExecutableProcessor.ExecuteExecutable(EngineExecutionPlan plan, IExecutable executable, ExecutionContext context)
at Microsoft.Practices.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()
at Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.ExecutableProcessor.ExecutePlan(EngineExecutionPlan plan, ExecutionContext context)
at Microsoft.Practices.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()
at Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.ExecutableProcessor.RunExecutables(EngineExecutionPlan plan, IList`1 executables)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.MultiStreamingExecutable.OnExecute(ISessionContext session) [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:28 AM [Thread:4328] [AbstractDataMovementExecutable`1:InformationEvent] (Info, Normal): DMS Manager finishing query: SID335, plan: 61ddf166-0d34-4537-b799-09b2e7f65aa8, queryId: QID510 [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:28 AM [Thread:4328] [MultiStreamingExecutable:InformationEvent] (Info, Normal): Early terminate other operations in stream. [Session.SessionId:SID335][Session.IsTransactional:False][Query.QueryId:QID510]
8/23/2019 12:28:28 AM [Thread:3812] [CommandWorker:WarningEvent] (Warning, Normal): Cannot cancel this Plan:61ddf166-0d34-4537-b799-09b2e7f65aa8, due to the uncancellable State:Completed
8/23/2019 12:28:28 AM [Thread:3812] [CommandWorker:InformationEvent] (Info, Normal): Cancel triggered for data movement
8/23/2019 12:28:28 AM [Thread:3652] [MultiStreamingExecutable:InformationEvent] (Info, Normal): One or more exceptions hit. Generating aggregated exception summary
===========================================
Exception summary
Exception Message: 107090;Query aborted-- the maximum reject threshold (1 rows) was reached while reading from an external source: 2 rows rejected out of total 2 rows processed.
Exception stack trace
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.AbstractDataMovementExecutable`1.HandleError()
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.AbstractDataMovementExecutable`1.OnExecute(ISessionContext Session)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.AbstractExecutable`1.Execute(ISessionContext session)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.ExecutableProcessor.<>c__DisplayClass40_0.<ExecuteExecutable>b__0(IList`1 propagationTokens)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.ExecutableProcessor.ExecuteActionInDistributedTransactions(Int32 transactionsCount, Action`1 action, IList`1 propagationTokens)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.ExecutableProcessor.ExecuteExecutable(EngineExecutionPlan plan, IExecutable executable, ExecutionContext context)
at Microsoft.Practices.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()
at Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
at Microsoft.SqlServer.DataWarehouse.Engine.Executables.ExecutableProcessor.ExecutePlan(EngineExecutionPlan plan, ExecutionContext context)
at Microsoft.Practices.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()
I would like know the reason for the error, which is not very clear from any of the error log files.