MVSFORUMS.com Forum Index MVSFORUMS.com
A Community of and for MVS Professionals
 
 FAQFAQ   SearchSearch   Quick Manuals   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

SPLIT Files Dynamically

 
Post new topic   Reply to topic   printer-friendly view    MVSFORUMS.com Forum Index -> Utilities
View previous topic :: View next topic  
Author Message
Sqlcode
Intermediate


Joined: 15 Dec 2006
Posts: 157
Topics: 38
PostPosted: Thu Sep 25, 2008 12:37 pm    Post subject: SPLIT Files Dynamically Reply with quote
Hi,
I am trying to split an input file into 10 equal output files.

I am using SPLITR utility of DFSORT to do the same.

Below the control card I am using.

Code:
//S1 EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN DD DSN=MY.INPUT.FILE
//T1 DD DSN=&&T1,UNIT=SYSDA,SPACE=(TRK,(1,1)),DISP=(,PASS)
//C1 DD DSN=&&C1,UNIT=SYSDA,SPACE=(TRK,(1,1)),DISP=(,PASS)
//CTL3CNTL DD *
OUTFIL FNAMES=(OUT01,OUT02,OUT03,OUT04,OUT05),
// DD DSN=*.C1,VOL=REF=*.C1,DISP=(OLD,PASS)
//OUT01 DD DSN=MY.OUTPUT.FILE01
//OUT02 DD DSN=MY.OUTPUT.FILE02
//OUT03 DD DSN=MY.OUTPUT.FILE03
//OUT04 DD DSN=MY.OUTPUT.FILE04
//OUT05 DD DSN=MY.OUTPUT.FILE05
//TOOLIN DD *
COPY FROM(IN) USING(CTL1)
COPY FROM(T1) TO(C1) USING(CTL2)
COPY FROM(IN) USING(CTL3)
/*
//CTL1CNTL DD *
OUTFIL FNAMES=T1,REMOVECC,NODETAIL,
TRAILER1=(COUNT=(M11,LENGTH=8))
/*
//CTL2CNTL DD *
OUTREC BUILD=(2X,C'SPLIT1R=',
1,8,ZD,DIV,+05,TO=ZD,LENGTH=8,80:X)
/*


Now,I also want to write limit records to one more output file.

Below is sample input and output records.

Code:
11111111
22222222
33333333
44444444
55555555
66666666
77777777
88888888
99999999
00000000
12345678
23456789
34567890
45678901
56789012


I am trying to split the same in 5 different output datasets. So that each 5 would like below

Code:
OUT01
11111111
22222222
33333333


Code:
OUT02
44444444
55555555
66666666



Code:
OUT03
77777777
88888888
99999999


Code:
OUT04
00000000
12345678
23456789


Code:
OUT05
34567890
45678901
56789012


Also, I want final output file with limit records as below

LIMITFILE
Code:
11111111         --> Start of File 1
33333333         --> End   of File 1
44444444         --> Start of File 2
66666666         --> End   of File 2

...
...
...
Code:
56789012         --> End   of File 5


I know I can dynamically split files using SPLITR but how to write Limit Records to output file.
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12372
Topics: 75
Location: San Jose
PostPosted: Thu Sep 25, 2008 3:31 pm    Post subject: Reply with quote
Sqlcode,


The following DFSORT JCL will give you the desired results

Code:

//STEP0100 EXEC PGM=ICETOOL                                 
//TOOLMSG  DD SYSOUT=*                                     
//DFSMSG   DD SYSOUT=*                                     
//IN       DD *                                             
11111111                                                   
22222222                                                   
33333333                                                   
44444444                                                   
55555555                                                   
66666666                                                   
77777777                                                   
88888888                                                   
99999999                                                   
00000000                                                   
12345678                                                   
23456789                                                   
34567890                                                   
45678901                                                   
56789012                                                   
//OUT01    DD SYSOUT=*                                     
//OUT02    DD SYSOUT=*                                     
//OUT03    DD SYSOUT=*                                     
//OUT04    DD SYSOUT=*                                     
//OUT05    DD SYSOUT=*                                     
//LIMIT    DD SYSOUT=*                                     
//CTL3CNTL DD DSN=&&C1,DISP=(,PASS),SPACE=(TRK,(1,1),RLSE) 
//TOOLIN DD *                                               
  COPY FROM(IN) USING(CTL1)                                 
  COPY FROM(T1) USING(CTL2)                                 
  COPY FROM(IN) USING(CTL3)                                 
/*                                                         
//CTL1CNTL DD *                                                 
  OUTFIL FNAMES=T1,REMOVECC,NODETAIL,                           
  TRAILER1=(COUNT=(M11,LENGTH=8))                               
/*                                                               
//CTL2CNTL DD *                                                 
  INREC OVERLAY=(10:1,8,ZD,DIV,+5,M11,LENGTH=8,X,               
                 +1,ADD,(+0,MUL,10,8,ZD),M11,LENGTH=8,X,         
                 +1,MUL,10,8,ZD,M11,LENGTH=8,X,                 
                 +1,ADD,(+1,MUL,10,8,ZD),M11,LENGTH=8,X,         
                 +2,MUL,10,8,ZD,M11,LENGTH=8,X,                 
                 +1,ADD,(+2,MUL,10,8,ZD),M11,LENGTH=8,X,         
                 +3,MUL,10,8,ZD,M11,LENGTH=8,X,                 
                 +1,ADD,(+3,MUL,10,8,ZD),M11,LENGTH=8,X,         
                 +4,MUL,10,8,ZD,M11,LENGTH=8,X,                 
                 +1,ADD,(+4,MUL,10,8,ZD),M11,LENGTH=8,X,         
                 +5,MUL,10,8,ZD,M11,LENGTH=8)                   
                                                                 
  OUTFIL FNAMES=CTL3CNTL,REMOVECC,NODETAIL,                           
  HEADER1=('  OUTREC OVERLAY=(81:SEQNUM,8,ZD)',/,                     
           '  OUTFIL FNAMES=(OUT01,OUT02,OUT03,OUT04,OUT05),',/,       
           '  SPLIT1R=',10,8,C',BUILD=(1,80)',/,                       
           '  OUTFIL FNAMES=LIMIT,',/,                                 
           '  INCLUDE=(81,8,ZD,EQ,',019,8,C',OR,',/,                   
           '           81,8,ZD,EQ,',028,8,C',OR,',/,                   
           '           81,8,ZD,EQ,',037,8,C',OR,',/,                   
           '           81,8,ZD,EQ,',046,8,C',OR,',/,                   
           '           81,8,ZD,EQ,',055,8,C',OR,',/,                   
           '           81,8,ZD,EQ,',064,8,C',OR,',/,                   
           '           81,8,ZD,EQ,',073,8,C',OR,',/,                   
           '           81,8,ZD,EQ,',082,8,C',OR,',/,                   
           '           81,8,ZD,EQ,',091,8,C',OR,',/,                   
           '           81,8,ZD,EQ,',100,8,C'),',/,                     
           '  IFOUTLEN=80,IFTHEN=(WHEN=INIT,BUILD=(1,10,',/,           
           '  SEQNUM,1,ZD,START=0,INCR=5)),',/,                       
           '  IFTHEN=(WHEN=(11,1,ZD,EQ,0),',/,                         
           '  OVERLAY=(11:C''','START OF FILE ''',',SEQNUM,1,ZD)),',/,
           '  IFTHEN=(WHEN=(11,1,ZD,EQ,5),',/,                         
           '  OVERLAY=(11:C''','END   OF FILE ''',                     
           ',SEQNUM,1,ZD))',80:X)                                     
/*


Hope this helps...

Cheers
_________________
Kolusu
www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Frank Yaeger
Sort Forum Moderator
Sort Forum Moderator


Joined: 02 Dec 2002
Posts: 1618
Topics: 31
Location: San Jose
PostPosted: Thu Sep 25, 2008 3:40 pm    Post subject: Reply with quote
Sqlcode,

Here's a DFSORT/ICETOOL job that will do what you asked for. I assumed your input file and output files have RECFM=FB and LRECL=80, but you can change the job appropriately for other attributes. The job uses DFSORT's new SUBSET and WHEN=GROUP functions available with z/OS DFSORT V1R5 PTF UK90013 (July, 2008).

Code:

//S1 EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN DD DSN=...  input file (FB/80)
//C1 DD DSN=&&C1,UNIT=SYSDA,SPACE=(TRK,(1,1)),DISP=(,PASS)
//C2 DD DSN=&&C2,UNIT=SYSDA,SPACE=(TRK,(1,1)),DISP=(,PASS)
//OUT01 DD DSN=...  output file1 (FB/80)
//OUT02 DD DSN=...  output file2 (FB/80)
//OUT03 DD DSN=...  output file3 (FB/80)
//OUT04 DD DSN=...  output file4 (FB/80)
//OUT05 DD DSN=...  output file5 (FB/80)
//LIMIT DD DSN=...  limit file (FB/80)
//TOOLIN DD *
SUBSET FROM(IN) TO(C1) OUTPUT KEEP LAST USING(CTL1)
COPY FROM(IN) USING(CTL2)
/*
//CTL1CNTL DD *
  INREC IFTHEN=(WHEN=INIT,BUILD=(SEQNUM,8,ZD)),
    IFTHEN=(WHEN=INIT,
       BUILD=(1,8,ZD,DIV,+5,TO=ZD,LENGTH=8))
  OUTFIL FNAMES=C1,BUILD=(C'  RECORDS=',1,8,C')',80:X)
  OUTFIL FNAMES=C2,BUILD=(C'  SPLIT1R=',1,8,80:X)
/*
//CTL2CNTL DD *
  INREC IFTHEN=(WHEN=GROUP,PUSH=(81:ID=1),
/*
//    DD DSN=*.C1,VOL=REF=*.C1,DISP=(OLD,PASS)
// DD *
  OUTFIL FNAMES=LIMIT,REMOVECC,NODETAIL,
    BUILD=(1,80),
    SECTIONS=(81,1,
      HEADER3=(1,8,5X,C'--> Start of File ',81,1),
      TRAILER3=(1,8,5X,C'--> End   of File ',81,1))
  OUTFIL FNAMES=(OUT01,OUT02,OUT03,OUT04,OUT05),
      BUILD=(1,80),
/*
//    DD DSN=*.C2,VOL=REF=*.C2,DISP=(OLD,PASS)


If you don't have z/OS DFSORT PTF UK90013, ask your System Programmer to install it (it's free).

For complete details on the new functions available with PTF UK90013, see:

www.ibm.com/systems/support/storage/software/sort/mvs/ugpf/
_________________
Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Sqlcode
Intermediate


Joined: 15 Dec 2006
Posts: 157
Topics: 38
PostPosted: Fri Sep 26, 2008 10:10 am    Post subject: Reply with quote
Frank,
Could you please briefly explain me logic behind this control card?

Here is what I understood:
CTL1 builds dynamic sort control card to split file into N dynamic output files. It uses RECORDS= N to decide when to start writting to new file. SPLIT1R decides records will be written to how many output files.

I can't understand logic behind CTL2. Here we create a GROUP but without any RECORDS or BEGIN condition!!!

I am trying to understand this control card so that if requirement changes I should be able to do it.

Thanks,
Back to top
View user's profile Send private message
Sqlcode
Intermediate


Joined: 15 Dec 2006
Posts: 157
Topics: 38
PostPosted: Fri Sep 26, 2008 11:30 am    Post subject: Reply with quote
Frank,
I get the logic after staring it for almost 50 minutes SmileSmileSmile.


But now I have one more question (though its not a requirement but trying to learn), How Do I align First record and last records for each section horizontally so that final LIMIT file would look like as below :

LIMITFILE
Code:
 
11111111         --> Start of File 1  33333333         --> End   of File 1
44444444         --> Start of File 2  66666666         --> End   of File 2
..
..
34567890         --> Start of File 5  56789012         --> End   of File 5

Can we merge TRAILER3 with HEADER3 as mentioned below :

Code:
SECTIONS=(81,1,
HEADER3=(1,8,5X,C'--> Start of File ',81,1,
                    TRAILER3=(1,8,5X,C'--> End   of File ',81,1)))

Logic here to get first record and last record for each section.

Thanks,
Back to top
View user's profile Send private message
Frank Yaeger
Sort Forum Moderator
Sort Forum Moderator


Joined: 02 Dec 2002
Posts: 1618
Topics: 31
Location: San Jose
PostPosted: Sat Sep 27, 2008 11:25 am    Post subject: Reply with quote
Quote:
I can't understand logic behind CTL2. Here we create a GROUP but without any RECORDS or BEGIN condition!!!


CTL2CNTL does have RECORDS=n - it comes from the C1 data set.

Quote:
Can we merge TRAILER3 with HEADER3 as mentioned below


No. HEADER3 writes a record before the data records of each section. Any fields you specify in HEADER3 are taken from the first record of the section. TRAILER3 writes a record after the data records of each section. Any fields you specify in TRAILER3 are taken from the last record of the section.

Combining them would require a different approach, or possibly another pass.
_________________
Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Sqlcode
Intermediate


Joined: 15 Dec 2006
Posts: 157
Topics: 38
PostPosted: Mon Sep 29, 2008 9:48 am    Post subject: Reply with quote
Frank,

Could you please help me out with combining start of record and end record?
OUT01-OUT05 would still be the same but LIMIT File will be as shown below :

LIMITFILE
Code:
011111111133333333
024444444466666666
...
...
...
053456789056789012


Here byte 1-2 shows OUT file's number, followed by start range and end range of LIMIT FILE.


Thanks,
Back to top
View user's profile Send private message
Frank Yaeger
Sort Forum Moderator
Sort Forum Moderator


Joined: 02 Dec 2002
Posts: 1618
Topics: 31
Location: San Jose
PostPosted: Mon Sep 29, 2008 12:58 pm    Post subject: Reply with quote
Assuming the limit values are ZD as shown in your example, you can replace SECTIONS with these DFSORT control statements:

Code:

    SECTIONS=(81,1,                                     
      TRAILER3=('0',81,1,MIN=(1,8,ZD,TO=ZD,LENGTH=8),   
       MAX=(1,8,ZD,TO=ZD,LENGTH=8)))                   

_________________
Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Back to top
View user's profile Send private message Send e-mail Visit poster's website
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12372
Topics: 75
Location: San Jose
PostPosted: Mon Sep 29, 2008 6:53 pm    Post subject: Reply with quote
Sqlcode,

Use the following DFSORT control cards which would give you the desired results for limit file even if it is character data.

Code:

//CTL1CNTL DD *                                         
  INREC IFTHEN=(WHEN=INIT,BUILD=(SEQNUM,8,ZD)),         
    IFTHEN=(WHEN=INIT,                                 
       BUILD=(1,8,ZD,DIV,+5,TO=ZD,LENGTH=8))           
  OUTFIL FNAMES=C1,BUILD=(C'  RECORDS=',1,8,C')',80:X) 
  OUTFIL FNAMES=C2,BUILD=(C'  SPLIT1R=',1,8,80:X)       
/*                                                     
//CTL2CNTL DD *                                         
  INREC IFTHEN=(WHEN=GROUP,PUSH=(81:ID=1,1,10),         
/*                                                     
//    DD DSN=*.C1,VOL=REF=*.C1,DISP=(OLD,PASS)         
// DD *                                                 
  OUTFIL FNAMES=LIMIT,REMOVECC,NODETAIL,BUILD=(80X),   
  SECTIONS=(81,1,                                       
  TRAILER3=(82,10,5X,                                   
            '--> START OF FILE ',81,1,5X,               
            1,10,                                       
            '--> END   OF FILE ',81,1,80:X))           
  OUTFIL FNAMES=(OUT01,OUT02,OUT03,OUT04,OUT05),       
      BUILD=(1,80),                                     
/*                                                     
//    DD DSN=*.C2,VOL=REF=*.C2,DISP=(OLD,PASS)         


The output from this is

Code:

11111111       --> START OF FILE 1     33333333  --> END   OF FILE 1
44444444       --> START OF FILE 2     66666666  --> END   OF FILE 2
77777777       --> START OF FILE 3     99999999  --> END   OF FILE 3
00000000       --> START OF FILE 4     23456789  --> END   OF FILE 4
34567890       --> START OF FILE 5     56789012  --> END   OF FILE 5



Hope this helps...

Cheers
_________________
Kolusu
www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Sqlcode
Intermediate


Joined: 15 Dec 2006
Posts: 157
Topics: 38
PostPosted: Mon Sep 29, 2008 8:40 pm    Post subject: Reply with quote
Wow....

Thanks a bunch Frank & Kolusu.
Back to top
View user's profile Send private message
Sqlcode
Intermediate


Joined: 15 Dec 2006
Posts: 157
Topics: 38
PostPosted: Tue Sep 30, 2008 3:46 pm    Post subject: Reply with quote
Frank & Kolusu,
When I ran the job with actual input dataset, it seems like not working as per required.

Input file has about 50000 records and I am trying to split that into 10 diffrent files. Here Total number of output file remains same. So I have modified SORTCARD to hardcode SPLIT1R=10(It did same without hardconding as well).

The intermediate SORT CARD generated is as shown below :
Code:
INREC IFTHEN=(WHEN=GROUP,PUSH=(81:ID=1),                           
RECORDS=00005000)                                                   
OUTFIL FNAMES=LIMIT,REMOVECC,NODETAIL,                             
  BUILD=(1,80),                                                     
    SECTIONS=(81,2,                                                 
        TRAILER3=(81,2,MIN=(1,8,ZD,TO=ZD,LENGTH=8 ),                 
         MAX=(1,8,ZD,TO=ZD,LENGTH=8 )))                             
OUTFIL FNAMES=(OUT01,OUT02,OUT03,OUT04,OUT05,OUT06,OUT07,OUT08,OUT09
               OUT10),BUILD=(1,80),                                                   
               SPLIT1R=10       



Since I intend to split input file in 10 output file, my SPLIT File should only have 10 records with start and end record but instead it has about 5K records. It also wrote only 10 records to each output file(OUT01 thru OUT09) and wrote rest of the records to OUT10(last output dataset).
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12372
Topics: 75
Location: San Jose
PostPosted: Tue Sep 30, 2008 4:05 pm    Post subject: Reply with quote
Sqlcode,

sqlcode wrote:
When I ran the job with actual input dataset, it seems like not working as per required.

Input file has about 50000 records and I am trying to split that into 10 diffrent files. Here Total number of output file remains same. So I have modified SORTCARD to hardcode SPLIT1R=10(It did same without hardconding as well).


SQLCODE,

if you are planning to split into 10 files then the SPLIT1R should be equal to 50,000/10 = 5,000 you have code just 10 and it is not going to work. Also since you have 10 files I suggest using ID=2 instead of 1 as it would have a proper counter.

Use the following control cards as IS which will give you the desired results

Code:

//CTL1CNTL DD *                                       
  INREC IFTHEN=(WHEN=INIT,BUILD=(SEQNUM,8,ZD)),       
    IFTHEN=(WHEN=INIT,                                 
       BUILD=(1,8,ZD,DIV,+10,TO=ZD,LENGTH=8))         
  OUTFIL FNAMES=C1,BUILD=(C'  RECORDS=',1,8,C')',80:X)
  OUTFIL FNAMES=C2,BUILD=(C'  SPLIT1R=',1,8,80:X)     
/*                                                     
//CTL2CNTL DD *                                       
  INREC IFTHEN=(WHEN=GROUP,PUSH=(81:ID=2,1,10),       
/*                                                     
//         DD DSN=*.C1,VOL=REF=*.C1,DISP=(OLD,PASS)   
//         DD *                                       
  OUTFIL FNAMES=LIMIT,REMOVECC,NODETAIL,BUILD=(80X),   
  SECTIONS=(81,2,                                     
  TRAILER3=(83,10,5X,                                 
            '--> START OF FILE ',81,2,5X,             
            1,10,                                     
            '--> END   OF FILE ',81,2,80:X))           
  OUTFIL FNAMES=(OUT01,OUT02,OUT03,OUT04,OUT05,       
                 OUT06,OUT07,OUT08,OUT09,OUT10),       
      BUILD=(1,80),                                   
/*                                                     
//         DD DSN=*.C2,VOL=REF=*.C2,DISP=(OLD,PASS)   

_________________
Kolusu
www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic   printer-friendly view    MVSFORUMS.com Forum Index -> Utilities All times are GMT - 5 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


MVSFORUMS
Powered by phpBB © 2001, 2005 phpBB Group