MVSFORUMS.com Forum Index MVSFORUMS.com
A Community of and for MVS Professionals
 
 FAQFAQ   SearchSearch   Quick Manuals   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Reducing Run Time in Sort

 
Post new topic   Reply to topic   printer-friendly view    MVSFORUMS.com Forum Index -> Utilities
View previous topic :: View next topic  
Author Message
Vilashini
Beginner


Joined: 10 Oct 2005
Posts: 2
Topics: 1
Location: Chennai
PostPosted: Mon Oct 10, 2005 6:48 am    Post subject: Reducing Run Time in Sort Reply with quote
Hi
I have the following steps in my PROC :

STEP 1 :
TASK A: Splitting an Input file of 10 million records into 3 files.
TASK B: Sort another two input files based on a condition.

STEP 2: PROGRAM STEP
STEP 3: Merging all the files from Step 1.

There is no problem with STEP 2. STEP 1 and STEP 3 were taking so much time (15 mins each) to complete as it needs a definite time optimisation. Is there any way by which the time could be reduced? Your help is greatly appreciated.
_________________
Vilashini
Back to top
View user's profile Send private message
Phantom
Data Mgmt Moderator
Data Mgmt Moderator


Joined: 07 Jan 2003
Posts: 1056
Topics: 91
Location: The Blue Planet
PostPosted: Mon Oct 10, 2005 7:06 am    Post subject: Reply with quote
Vilashini,

We need to first look at the JCL used so that we can give some suggestion. Post your complete JCL. If your SORT card is stored in a PDS member or PS, then please post that too.

PS: When you paste any code, please use BB Tags {code} & {/code} as shown below.
{code}
line 1
line 2
line 3
line 4
....
{/code}

Remember, You need to change '{' by '[' and '}' by ']' in the above lines.

Thanks,
Phantom
Back to top
View user's profile Send private message
Vilashini
Beginner


Joined: 10 Oct 2005
Posts: 2
Topics: 1
Location: Chennai
PostPosted: Mon Oct 10, 2005 7:26 am    Post subject: Reply with quote
Code:

//R010     EXEC PGM=SYNCTOOL,PARM='PARASORT'                   
//IN1      DD  DSN=FILE1,                 
//             DISP=SHR                                       
//IN2      DD  DSN=FILE2,                     
//             DISP=SHR                                       
//IN3      DD  DSN=FILE3,       
//             DISP=SHR                                       
//OUT1     DD  DSN=FILE.OUT1,
//             DISP=(,CATLG,DELETE),
//             UNIT=SYSDA,                                     
//             STORCLAS=SCRWARP,                               
//             DATACLAS=COMPRESS,                             
//             AVGREC=K,                                       
//             SPACE=(1060,(6700,6000),RLSE),                 
//             DCB=(RECFM=FB,LRECL=1060)                       
//OUT11    DD  DSN=FILE1.OUT11,                 
//             DISP=(,CATLG,DELETE),                           
//             UNIT=SYSDA,       
//             STORCLAS=SCRWARP,                 
//             DATACLAS=COMPRESS,                 
//             AVGREC=K,                         
//             SPACE=(1060,(3700,3000),RLSE),     
//             DCB=(RECFM=FB,LRECL=1060)         
//OUT12    DD  DSN=FILE1.OUT12,   
//             DISP=(,CATLG,DELETE),             
//             UNIT=SYSDA,                       
//             STORCLAS=SCRWARP,                 
//             DATACLAS=COMPRESS,                 
//             AVGREC=K,                         
//             SPACE=(1000,(8700,8000),RLSE),     
//             DCB=(RECFM=FB,LRECL=1000)         
//OUT2     DD  DSN=FILE2,     
//             DISP=(,CATLG,DELETE),             
//             UNIT=SYSDA,                       
//             STORCLAS=SCRWARP,                 
//             DATACLAS=COMPRESS, 
//             AVGREC=K,                         
//             SPACE=(1860,(7700,7000),RLSE),   
//             DCB=(RECFM=FB,LRECL=1860)         
//OUT3     DD  DSN=FILE3,
//             DISP=(,CATLG,DELETE),             
//             UNIT=SYSDA,                       
//             STORCLAS=SCRWARP,                 
//             DATACLAS=COMPRESS,               
//             AVGREC=K,                         
//             SPACE=(3560,(3000,1000),RLSE),   
//             DCB=(RECFM=FB,LRECL=3560) 
//TOOLIN   DD  *
  SORT FROM(IN1) USING(CTL1)
  SORT FROM(IN1) TO(OUT12) USING(CTL2)
  SORT FROM(IN2) TO(OUT2) USING(CTL3)
  SORT FROM(IN3) TO(OUT3) USING(CTL4)
/*
//CTL1CNTL DD  *
  SORT IELDS=(9,9,CH,A,48,1,CH,A,35,9,CH,A,
              91,6,CH,A,946,20,CH,A,116,1,CH,A,
              293,10,CH,A,163,10,CH,A)
  OUTFIL FNAMES=OUT1,
  INCLUDE=(740,1,CH,EQ,C'P',AND,115,1,CH,EQ,C'T'),
  OUTREC=(1,1000,9,9,48,1,35,9,91,6,14X,116,1,293,10,163,10)
  OUTFIL FNAMES=OUT11,
  INCLUDE=(740,1,CH,EQ,C'G',AND,115,1,CH,EQ,C'T'),
  OUTREC=(1,1000,9,9,48,1,35,9,946,20,116,1,293,10,163,10)
/*
//CTL2CNTL DD *                             
  SORT FIELDS=COPY
  INCLUDE COND=((740,1,CH,NE,C'G',OR,740,1,CH,NE,C'P'),
      AND,(115,1,CH,NE,C'T'))
/*
//CTL3CNTL DD  *
  SORT FIELDS=(17,9,CH,A,33,1,CH,A,1,9,CH,A,
               79,6,CH,A,49,1,CH,A,60,10,PD,A,
               50,10,PD,A)
  OUTREC FIELDS=(1,1800,17,9,33,1,1,9,79,6,14X,49,1,60,10,50,10)
/*
//CTL4CNTL DD  *
  SORT FIELDS=(93,9,CH,A,109,1,CH,A,77,9,CH,A,
              198,20,CH,A,125,1,CH,A,136,10,PD,A,
              126,10,PD,A)
  OUTREC FIELDS=(1,3500,93,9,109,1,77,9,198,20,125,1,136,10,126,10)
/*

_________________
Vilashini
Back to top
View user's profile Send private message
SureshKumar
Intermediate


Joined: 23 Jan 2003
Posts: 211
Topics: 21
PostPosted: Mon Oct 10, 2005 8:06 am    Post subject: Reply with quote
vilashini,
10 million + records and the amount of sort work in a single step 15 minutes is not surprising - is it the elapsed time that's of concern or the CPU?. Try increasing the VSCORET to 128M (ask around for limits). Look for //$ORTPARM card on other sort Jobs, This helps but not a lot. Thanks
Back to top
View user's profile Send private message
Cogito-Ergo-Sum
Advanced


Joined: 15 Dec 2002
Posts: 637
Topics: 43
Location: Bengaluru, INDIA
PostPosted: Mon Oct 10, 2005 8:27 am    Post subject: Reply with quote
Why not split the TOOLIN into different steps? Each such step can be run in parallel job.
_________________
ALL opinions are welcome.

Debugging tip:
When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.
-- Sherlock Holmes.
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12376
Topics: 75
Location: San Jose
PostPosted: Mon Oct 10, 2005 8:46 am    Post subject: Reply with quote
Vilasini,

A couple of recommedations .

1.You CANNOT pass the parm "parasort" to synctool. Synctool does not have any capability of receiving parms. You need to pass parms via DFSPARM.

2.The first sort and second pass can be clubbed in to one sort by using the parm save.

i.E
Code:

SORT FROM(IN1) USING(CTL1)
SORT FROM(IN1) TO(OUT12) USING(CTL2)


TO

Code:

SORT FROM(IN1) USING(CTL1)


Code:


//CTL1CNTL DD  *
  SORT FIELDS=(09,09,CH,A,
               48,01,CH,A,
               35,09,CH,A,
               91,06,CH,A,
              946,20,CH,A,
              116,01,CH,A,
              293,10,CH,A,
              163,10,CH,A)
  OUTFIL FNAMES=OUT1,
  INCLUDE=(740,1,CH,EQ,C'P',AND,115,1,CH,EQ,C'T'),
  OUTREC=(1,1000,9,9,48,1,35,9,91,6,14X,116,1,293,10,163,10)
  OUTFIL FNAMES=OUT11,
  INCLUDE=(740,1,CH,EQ,C'G',AND,115,1,CH,EQ,C'T'),
  OUTREC=(1,1000,9,9,48,1,35,9,946,20,116,1,293,10,163,10)
  OUTFIL FNAMES=OUT12,SAVE
/*


Hope this helps...

Cheers

Kolusu
_________________
Kolusu
www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Phantom
Data Mgmt Moderator
Data Mgmt Moderator


Joined: 07 Jan 2003
Posts: 1056
Topics: 91
Location: The Blue Planet
PostPosted: Mon Oct 10, 2005 9:31 am    Post subject: Reply with quote
Vilashini,

Kolusu's method should save you considerable time since you are eliminating 1 full pass. As Kolusu mentioned use the PARASORT in DFSPARM as shown below.

Code:

//TOOLIN   DD  *
SORT FROM(IN1) USING(CTL1)
SORT FROM(IN2) TO(OUT2) USING(CTL3)
SORT FROM(IN3) TO(OUT3) USING(CTL4)
/*
//DFSPARM DD *           
  PARASORT               
/*                       
//CTL1CNTL DD  *
  SORT FIELDS=(09,09,CH,A,
               48,01,CH,A,
               35,09,CH,A,
               91,06,CH,A,
              946,20,CH,A,
              116,01,CH,A,
              293,10,CH,A,
              163,10,CH,A)
  OUTFIL FNAMES=OUT1,
  INCLUDE=(740,1,CH,EQ,C'P',AND,115,1,CH,EQ,C'T'),
  OUTREC=(1,1000,9,9,48,1,35,9,91,6,14X,116,1,293,10,163,10)
  OUTFIL FNAMES=OUT11,
  INCLUDE=(740,1,CH,EQ,C'G',AND,115,1,CH,EQ,C'T'),
  OUTREC=(1,1000,9,9,48,1,35,9,946,20,116,1,293,10,163,10)
  OUTFIL FNAMES=OUT12,SAVE
/*


Kolusu,
Quote:

1.You CANNOT pass the parm "parasort" to synctool. Synctool does not have any capability of receiving parms. You need to pass parms via DFSPARM.


I didn't know this. Thanks for the update.

Regards,
Phantom
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12376
Topics: 75
Location: San Jose
PostPosted: Mon Oct 10, 2005 5:11 pm    Post subject: Reply with quote
vilashini,

You don't have to code DCB parameters for SOrt. Sort will automatically calculate the DCB parameters from the input or inrec/outrec statements. Also change your output file allocation to cylinders instead of blocks.

Also I see that you are compressing and stripping the output datasets (dataclas and storclas parameters). Remember that with compression and stripping your Cpu time will be high.

Code your control cards in orderly fashion with comments so that it is easy to maintain. Ex: look at the control cards in CTL1.



This is how your JCL should look like.

Code:

//STEP0100 EXEC PGM=SYNCTOOL
//TOOLMSG  DD  SYSOUT=*
//DFSMSG   DD  SYSOUT=*
//IN1      DD  DSN=FILE1,                 
//             DISP=SHR                                       
//IN2      DD  DSN=FILE2,                     
//             DISP=SHR                                       
//IN3      DD  DSN=FILE3,       
//             DISP=SHR                                       
//OUT1     DD  DSN=FILE.OUT1,                   
//             DISP=(NEW,CATLG,DELETE),                           
//             UNIT=SYSDA,                                     
//             STORCLAS=SCRWARP,                               
//             DATACLAS=COMPRESS,                             
//             SPACE=(CYL,(642,128),RLSE)                 
/*
//OUT11    DD  DSN=FILE1.OUT11,                 
//             DISP=(NEW,CATLG,DELETE),                           
//             UNIT=SYSDA,       
//             STORCLAS=SCRWARP,                 
//             DATACLAS=COMPRESS,                 
//             SPACE=(CYL,(642,128),RLSE)     
/*
//OUT12    DD  DSN=FILE1.OUT12,   
//             DISP=(NEW,CATLG,DELETE),             
//             UNIT=SYSDA,                       
//             STORCLAS=SCRWARP,                 
//             DATACLAS=COMPRESS,                 
//             SPACE=(CYL,(618,123),RLSE)     
/*         
//OUT2     DD  DSN=FILE2,     
//             DISP=(NEW,CATLG,DELETE),             
//             UNIT=SYSDA,                       
//             STORCLAS=SCRWARP,                 
//             DATACLAS=COMPRESS, 
//             SPACE=(CYL,(1112,222),RLSE)
/*       
//OUT3     DD  DSN=FILE3,
//             DISP=(NEW,CATLG,DELETE),             
//             UNIT=SYSDA,                       
//             STORCLAS=SCRWARP,                 
//             DATACLAS=COMPRESS,               
//             SPACE=(CYL,(2381,476),RLSE)
/*
//TOOLIN   DD  *
  SORT FROM(IN1) USING(CTL1)
  SORT FROM(IN2) TO(OUT2) USING(CTL2)
  SORT FROM(IN3) TO(OUT3) USING(CTL3)
/*
//DFSPARM DD *           
  PARASORT               
/*       
//CTL1CNTL DD  *
  OPTION DYNALLOC              $ DYNAMICALLY ALLOCATE SORTWK
  SORT FIELDS=(009,09,CH,A,    $ SORT FIELD-1               
               048,01,CH,A,    $ SORT FIELD-2               
               035,09,CH,A,    $ SORT FIELD-3               
               091,06,CH,A,    $ SORT FIELD-4               
               946,20,CH,A,    $ SORT FIELD-5               
               116,01,CH,A,    $ SORT FIELD-6               
               293,10,CH,A,    $ SORT FIELD-7               
               163,10,CH,A)    $ SORT FIELD-8               
                                                             
  OUTFIL FNAMES=OUT1,                                       
  INCLUDE=(740,1,CH,EQ,C'P',AND,                             
           115,1,CH,EQ,C'T'),                               

  OUTREC=(001,1000,            $ FIRST 1000 BYTES           
          009,0009,            $ COPY 9 BYTES FROM POS 9     
          048,0001,            $ COPY 1 BYTES FROM POS 48   
          035,0009,            $ COPY 9 BYTES FROM POS 35   
          091,0006,            $ COPY 6 BYTES FROM POS 91   
          14X,                 $ 14 SPACES                   
          116,0001,            $ COPY 1 BYTES FROM POS 116   
          293,0010,            $ COPY 10 BYTES FROM POS 293 
          163,0010)            $ COPY 10 BYTES FROM POS 163 

  OUTFIL FNAMES=OUT11,
  INCLUDE=(740,1,CH,EQ,C'G',AND,
           115,1,CH,EQ,C'T'),
  OUTREC=(001,1000,
          009,0009,
          048,0001,
          035,0009,
          946,0020,
          116,0001,
          293,0010,
          163,0010)

  OUTFIL FNAMES=OUT12,SAVE
/*
//CTL2CNTL DD  *
  OPTION DYNALLOC
  SORT FIELDS=(17,09,CH,A,
               33,01,CH,A,
               01,09,CH,A,
               79,06,CH,A,
               49,01,CH,A,
               60,10,PD,A,
               50,10,PD,A)

  OUTREC FIELDS=(01,1800,
                 17,0009,
                 33,0001,
                 01,0009,
                 79,0006,
                 14X,
                 049,0001,
                 60,0010,
                 50,0010)
                           
/*
//CTL3CNTL DD  *
  OPTION DYNALLOC
  SORT FIELDS=(093,09,CH,A,
               109,01,CH,A,
               077,09,CH,A,
               198,20,CH,A,
               125,01,CH,A,
               136,10,PD,A,
               126,10,PD,A)

  OUTREC FIELDS=(001,3500,
                 093,0009,
                 109,0001,
                 077,0009,
                 198,0020,
                 125,0001,
                 136,0010,
                 126,0010)
/*


Hope this helps...

Cheers

Kolusu
_________________
Kolusu
www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Phantom
Data Mgmt Moderator
Data Mgmt Moderator


Joined: 07 Jan 2003
Posts: 1056
Topics: 91
Location: The Blue Planet
PostPosted: Tue Oct 11, 2005 1:16 am    Post subject: Reply with quote
Kolusu,

Quote:

Also I see that you are compressing and stripping the output datasets (dataclas and storclas parameters). Remember that with compression and stripping your Cpu time will be high.


I agree with you. But Thatz pretty unfortunate. I know those files (my previous shop). They are very huge and without compression, it ends in space problem (either lack of space / lack of volumes to allocate huge space).

I believe that with the correction of PARM (PARASORT), Vilashini should see some significant difference in the run time.

Thanks,
Phantom
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic   printer-friendly view    MVSFORUMS.com Forum Index -> Utilities All times are GMT - 5 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


MVSFORUMS
Powered by phpBB © 2001, 2005 phpBB Group