Showing posts with label task. Show all posts
Showing posts with label task. Show all posts

Tuesday, March 27, 2012

can I get the Parent task in a custom data flow task?

I'm building a custom data flow task to handle errors in a generic way in C#. I want to output the Task name (the name of the Data flow) as well as the Task name that is inputting data into my custom task.

Is there a way to get both the task name and the previous data flow task name that's inputting the data?

THanks in advance,
-Chris

PS, the VariableContainer didn't seem to contain any of the system variables- ex. the below statement returned false.

this.VariableDispenser.Contains("System::SourceName")

Are you talking about a control flow task or a data flow component? For a dataflow component you can not get any information about any other component other than yourself.

As for the variable dispenser, unless you have added variables to it via the lock methods then it contains no variables. Additionally, there is no such system variable as SourceName.

Thanks,
Matt|||I am talking about a dataflow component.

Do I have to add the system variables to the VariableDispenser?

Thanks for your help Matt,
-Chris|||Yes, you have to add all variables to the variable dispenser in the form of the lock methods. Otherwise the variable dispenser is empty.

Matt|||

I added the System::SourceName variable to the custom data flow component by using LockOneForWrite, but gave an error that that variable doesn't exist.

Is there a place where I can find a listing of the system variables available at the data flow component level?

|||As I mentioned in my original post, there is no such system variable. You can look in the designer to see all the system variable that are available.

Matt|||

Matt David wrote:

As I mentioned in my original post, there is no such system variable. You can look in the designer to see all the system variable that are available.

Matt

Just to clarify Matt's point...SSIS actually does have a System::SourceName variable but it is only available in the eventhandler container. You can't reference it in your control-flow.

-Jamie|||I believe you can get hold of the component next to it in the flow, by looking at one of the input collections. However you can't find out the component that added a column to a flow.

can I get the Parent task in a custom data flow task?

I'm building a custom data flow task to handle errors in a generic way in C#. I want to output the Task name (the name of the Data flow) as well as the Task name that is inputting data into my custom task.

Is there a way to get both the task name and the previous data flow task name that's inputting the data?

THanks in advance,
-Chris

PS, the VariableContainer didn't seem to contain any of the system variables- ex. the below statement returned false.

this.VariableDispenser.Contains("System::SourceName")

Are you talking about a control flow task or a data flow component? For a dataflow component you can not get any information about any other component other than yourself.

As for the variable dispenser, unless you have added variables to it via the lock methods then it contains no variables. Additionally, there is no such system variable as SourceName.

Thanks,
Matt|||I am talking about a dataflow component.

Do I have to add the system variables to the VariableDispenser?

Thanks for your help Matt,
-Chris|||Yes, you have to add all variables to the variable dispenser in the form of the lock methods. Otherwise the variable dispenser is empty.

Matt|||

I added the System::SourceName variable to the custom data flow component by using LockOneForWrite, but gave an error that that variable doesn't exist.

Is there a place where I can find a listing of the system variables available at the data flow component level?

|||As I mentioned in my original post, there is no such system variable. You can look in the designer to see all the system variable that are available.

Matt|||

Matt David wrote:

As I mentioned in my original post, there is no such system variable. You can look in the designer to see all the system variable that are available.

Matt

Just to clarify Matt's point...SSIS actually does have a System::SourceName variable but it is only available in the eventhandler container. You can't reference it in your control-flow.

-Jamie|||I believe you can get hold of the component next to it in the flow, by looking at one of the input collections. However you can't find out the component that added a column to a flow.

Thursday, February 16, 2012

can a query be written to do this...

I have the following table defined: Not my design/idea and I can't change
it)
CREATE TABLE [dbo].[Task] (
[TaskID] [ROWIDENTIFIER] NOT NULL ,
[Name] [SHORTNAME] NULL ,
[Description] [SHORTDESCRIPTION] NULL ,
[PreviousTaskID] [ROWIDENTIFIER] NULL ,
[NextTaskID] [ROWIDENTIFIER] NULL ,
[ProcedureID] [ROWIDENTIFIER] NOT NULL ,
[IsActive] [WFBOOL] NULL
) ON [PRIMARY]
PreviousTaskID and NextTaskID form, what amounts to a linked list where
PreviousTaskID points to the TaskID that comes before the current task and
NextTaskID points to the TaskID of the task that follows the current task.
A PreviousTaskID equal to null signifies the first task in a list and a
NextTaskID equal to null signifies it is the last task in the list.
With all of that in mind: Is there any way to write a query that will return
a single set of rows ordered from first to last?
TIA
Brian WBW -
It seems unnecessary to have a Next and Previous so long as the chain is
always 1 for 1 (i.e. Task 2 always comes after Task 1, etc). Anyways, here'
s
something that should get you started:
create table #Task (
TaskID int not null
, [Name] varchar(50) null
, PreviousTaskID int null
, NextTaskID int null
)
insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(1,
'T1', null, 2)
insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(2,
'T1', 1, 3)
insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(3,
'T1', 2, 4)
insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(4,
'T1', 3, 5)
insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(5,
'T1', 4, 6)
insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(6,
'T1', 5, null)
declare @.parent_level int
set @.parent_level = 0
declare @.hierarchy table (parent int, item int, [level] int)
insert into @.hierarchy (parent, item, [level])
select null, taskid, 0
from #Task
where previoustaskid is null
while 1 = 1
begin
insert into @.hierarchy(parent, item, [level])
select nexttaskid, taskid, @.parent_level + 1
from #Task
where previoustaskid in (select item from @.hierarchy where [level] =
@.parent_level)
if @.@.rowcount = 0
break
set @.parent_level = @.parent_level + 1
end
select t.TaskID, h.[level] as Ordering
from #Task t
join @.hierarchy h on t.TaskID = h.item
order by 2 asc|||Perfect!
muchos gracias!
"Cris_Benge" <CrisBenge@.discussions.microsoft.com> wrote in message
news:C3096324-4209-4868-80AB-FA4FA7DB9B97@.microsoft.com...
> BW -
> It seems unnecessary to have a Next and Previous so long as the chain is
> always 1 for 1 (i.e. Task 2 always comes after Task 1, etc). Anyways,
here's
> something that should get you started:
> create table #Task (
> TaskID int not null
> , [Name] varchar(50) null
> , PreviousTaskID int null
> , NextTaskID int null
> )
> insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(1,
> 'T1', null, 2)
> insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(2,
> 'T1', 1, 3)
> insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(3,
> 'T1', 2, 4)
> insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(4,
> 'T1', 3, 5)
> insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(5,
> 'T1', 4, 6)
> insert into #Task (TaskID, [Name], PreviousTaskId, NextTaskID) values(6,
> 'T1', 5, null)
> declare @.parent_level int
> set @.parent_level = 0
> declare @.hierarchy table (parent int, item int, [level] int)
> insert into @.hierarchy (parent, item, [level])
> select null, taskid, 0
> from #Task
> where previoustaskid is null
> while 1 = 1
> begin
> insert into @.hierarchy(parent, item, [level])
> select nexttaskid, taskid, @.parent_level + 1
> from #Task
> where previoustaskid in (select item from @.hierarchy where [level] =
> @.parent_level)
> if @.@.rowcount = 0
> break
> set @.parent_level = @.parent_level + 1
> end
> select t.TaskID, h.[level] as Ordering
> from #Task t
> join @.hierarchy h on t.TaskID = h.item
> order by 2 asc
>

Tuesday, February 14, 2012

Can a Data Flow Task mimic Bulk Insert?

Hi Guys,

The way I understand a Data Flow Task is that it inserts the rows from the source to destination one by one. Is there a way to make it act like a bulk insert task? We have been experiencing performance issues when inserting a lot of rows from one table to another. If there's no way to actually do it, can a bulk insert task functionality be scripted? Coz what I need is a table to table insert, and the bulk insert task only accepts data files as sources.

Thanks!

Kervy

What destination object are you using? The OLEDB Destination and the SQL Server destination both enable bulk insert options for SQL Server.

How many rows are you inserting?

Donald

|||

Oh you mean to say, by default, an OLEDB source inserts to an OLEDB Destination in bulk? We are only inserting a couple of hundred thousand rows, but the source table is in the US. destinations are in Europe and Asia Pacific, so we need to optimize this as much as we can given the proximity of the servers.

We are using OLEDB source and OLEDB Destination

Thanks

|||

If you need to performance tune your data-flow then you should read a whitepaper that I have mentioned here: http://blogs.conchango.com/jamiethomson/archive/2006/04/09/3594.aspx

There's also more useful information here: http://blogs.conchango.com/jamiethomson/archive/2005/04/20/1319.aspx

Note that SSIS does not insert records one at a time - that would be a performance killer. The SSIS pipeline processes data in buffers and the contents of each buffer is inserted at the same time! A large part of performance tuning your application is finding the optimum buffer size.

Note that in your scenario the bottleneck is likely to be the georgraphic disparity of the source and destination, not how fast SSIS can process it. Try and reduce the amount of data that is coming over the wire if possible - i.e. Filter the data at source and only select columns that you require.

-Jamie

|||

hi Jamie,

Thanks for that response. I will read more on how to performance tune my data flow. One accepted cause for the horrible execution time is the distance between servers (destinations are in 2 different continents!), that is why we are trying to figure out if there are other ways to improve on this. We tried testing with a destination server in the same area, and it took about 9 minutes.. compared to the 16 hours it took when the target was Asia Pacific.. we may have to accept that performance issue for now..

total number of rows to be inserted = 200,000

total number of columns = 167

|||

KervyChoa wrote:

hi Jamie,

Thanks for that response. I will read more on how to performance tune my data flow. One accepted cause for the horrible execution time is the distance between servers (destinations are in 2 different continents!), that is why we are trying to figure out if there are other ways to improve on this. We tried testing with a destination server in the same area, and it took about 9 minutes.. compared to the 16 hours it took when the target was Asia Pacific.. we may have to accept that performance issue for now..

total number of rows to be inserted = 200,000

total number of columns = 167

Exactly, hence my previous comment "Try and reduce the amount of data that is coming over the wire if possible - i.e. Filter the data at source and only select columns that you require."

The point here is to reduce the amount of data coming over the wire because that is quite obviously the bottleneck in your scenario.

-Jamie

|||I tried using perfmon to see what can be done, but I can't find the SQL Server: SSIS Pipeline Performance object in perfmon.. Am I missing something here?|||

KervyChoa wrote:

I tried using perfmon to see what can be done, but I can't find the SQL Server: SSIS Pipeline Performance object in perfmon.. Am I missing something here?

As we have already determined (twice) the bottleneck is not within the SSIS pipeline so looking at this perf counter is not going to help you.

Unless you pick one of the machines up and move it to a different continent, or improve the speed of the link between them, you're not going to see any eprformance improvement.

-Jamie

Friday, February 10, 2012

Calling vb script from SQL job

I am trying to execute a visual basic script using a SQL job. I can schedul
e
this as a scheduled task and then use a separate SQL job to compelte the
formatting tasks, but prefer to keep all steps consolidated so it is easier
to trouble shoot.
I tried using xp_cmdshell 'job.vbs', no_output
but it gives the error that this is not a recognized internal command.
Any help would be greatly appreciated.>I am trying to execute a visual basic script using a SQL job. I can
>schedule
> this as a scheduled task and then use a separate SQL job to compelte the
> formatting tasks,
Why don't you have the VBScript task call a stored procedure?
A|||Hello Derekman:
You wrote on Mon, 26 Jun 2006 12:12:03 -0700:
D> I am trying to execute a visual basic script using a SQL job. I can
D> schedule this as a scheduled task and then use a separate SQL job to
D> compelte the formatting tasks, but prefer to keep all steps consolidated
D> so it is easier to trouble shoot.
D> I tried using xp_cmdshell 'job.vbs', no_output
D> but it gives the error that this is not a recognized internal command.
You can run only an executable; sql server won't look into the registry to
see what is the application handling file extension. In Explorer, this is
most likely specified as
WScript.exe "%1" %*
That's how it should be in sql job as well.
Vadim Rapp